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Preface 


This  report  presents  the  results  of  two  RAND  studies.  One  compares  changes  in 
requirements,  technical  risk,  and  cost  growth  encountered  in  the  development  of  the 
key  software  products  produced  by  the  Joint  Tactical  Radio  System  (JTRS)  network 
enterprise  domain  (NED)  program  with  similar  elements  for  the  commercial  cell  phone 
fourth-generation  (4G)  long-term  evolution  (LTE)  waveform.  The  study  examines  how 
JTRS  networking  waveforms’  technical  and  associated  program  risks  were  managed, 
the  cost  of  developing  JTRS  networking  waveforms,  and  how  technical  risks  and  chal¬ 
lenging  requirements  contributed  to  schedule  and  cost  increases.  The  objective  of  this 
effort  was  to  identify  differences  between  the  program  management  factors  and  cost 
structures  and  investigate  the  underlying  causes  of  these  differences.  The  second  study 
analyzes  programs  that  have  had  multiple  Nunn-McCurdy  breaches  (that  is,  military 
acquisition  programs  that  have  exceeded  certain  cost  thresholds)  in  an  attempt  to  iden¬ 
tify  characteristics  of  programs  that  overrun  their  budgets. 

This  research  was  sponsored  by  the  Performance  Assessments  and  Root  Cause 
Analysis  (PARCA)  office,  in  the  Office  of  the  Assistant  Secretary  of  Defense  for  Acqui¬ 
sition,  and  conducted  within  the  Acquisition  and  Technology  Policy  Center  of  the 
RAND  National  Defense  Research  Institute,  a  federally  funded  research  and  develop¬ 
ment  center  sponsored  by  the  Office  of  the  Secretary  of  Defense,  the  Joint  Staff,  the 
Unified  Combatant  Commands,  the  Navy,  the  Marine  Corps,  the  defense  agencies, 
and  the  defense  Intelligence  Community. 

For  more  information  on  the  RAND  Acquisition  and  Technology  Policy  Center, 
see  http://www.rand.org/nsrd/ndri/centers/atp.html  or  contact  the  director  (contact 
information  is  provided  on  the  web  page). 
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Summary 


This  report  presents  the  results  of  two  RAND  studies:  One  is  a  comparative  assessment 
of  the  capabilities  and  development  approaches  used  for  the  Department  of  Defense 
(DoD)  Joint  Tactical  Radio  System  (JTRS)  wideband  networking  waveform  (WNW) 
(a  key  component  of  a  planned  tactical  military  communications  system)  and  the  long¬ 
term  evolution  (LTE)  waveform  (a  key  component  of  advanced  commercial  mobile 
communications  networks).  The  second  study  analyzes  acquisition  programs  with 
multiple  Nunn-McCurdy  breaches,  that  is,  military  acquisition  programs  that  have 
exceeded  certain  cost  thresholds. 

The  first  study  compares  differences  in  system  designs,  technical  requirements, 
intellectual  property  protection  schemes,  and  cost  in  the  development  of  WNW,  one  of 
the  key  software  products  produced  by  the  JTRS  network  enterprise  domain1  (NED) 
program  with  similar  elements  of  the  commercial  cell  phone  fourth-generation  (4G) 
LTE  waveform.  The  study  first  examined  how  the  program  managed  system  design, 
technical  risks,  and  WNW  development  costs  and  how  technical  risks  and  challenging 
requirements  contributed  to  schedule  and  cost  increases.  These  JTRS  NED  program 
management  factors  were  then  compared  to  the  risk  factors,  program  management 
approaches,  schedule  changes,  and  development  costs  for  the  LTE  waveform,  the  most 
advanced  wireless  waveform  ever  developed  by  commercial  industry. 

The  second  study  analyzes  programs  that  have  had  multiple  Nunn-McCurdy 
breaches  in  an  attempt  to  identify  unique  characteristics  of  programs  that  overrun 
their  budgets. 

The  main  findings  and  recommendations  of  both  efforts  are  described  below. 


JTRS  NED  Findings  and  Recommendations 

Looking  across  the  issues  with  the  JTRS  GMR  WNW  program  that  most  contribute 
to  its  less-than-favorable  comparison  with  LTE,  four  stand  out.  First,  the  amount  spent 
on  JTRS  Ground  Mobile  Radio  (GMR)  and  WNW  research,  development,  test,  and 
evaluation  (RDT&E)  appears  to  be  far  less  than  was  needed  when  compared  to  the 


1  Essentially  a  system  of  systems. 
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amount  spent  for  RDT&E  by  commercial  firms  in  the  development  of  LTE.  Second, 
the  more  evolutionary  development  approach  for  LTE  along  with  the  higher  RDT&E 
expenditure  enabled  project  personnel  to  deal  with  system  performance  and  technol¬ 
ogy  risk  issues  as  they  came  up.  Third,  the  JTRS  program  structure  that  separated 
hardware  and  software  development  coupled  with  a  “big  bang”  acquisition  approach 
(the  delivery  of  a  system  that  could  meet  all  requirements  at  Milestone  C)  not  only 
complicated  coordination  needs  among  programs  but  also  may  have  delayed  the  dis¬ 
covery  of  integration  problems  until  it  was  too  late  or  too  near  Milestone  C.  And 
last,  the  intellectual  property  (IP)  model  used  in  the  DoD  JTRS  program  may  have 
prevented  the  incorporation  of  needed  technologies  into  the  program  that  could  have 
reduced  technical  risks  and  performance  issues.  In  contrast,  the  more  inclusive  Third- 
Generation  Partnership  Project  (3GPP)  IP  model  encouraged  the  incorporation  of  a 
worldwide  and  world-class  set  of  technology  IP  into  LTE. 

Findings 

Organizations  and  Intellectual  Property  Rights  Differ  Substantially 

Although  the  LTE  and  the  JTRS  GMR  WNW  differ  significantly  in  many  techni¬ 
cal  and  design  aspects,  they  share  some  common  underlying  wireless  communications 
technologies. 

The  organizational  structures  that  spearheaded  the  development  of  these  two 
systems  are  quite  different.  Furthermore,  the  LTE  and  GMR  WNW  development 
approaches  and  architectures  are  significantly  different.  The  LTE  architecture  is  not 
software-defined,  but  hardware  components  (microchips  in  particular)  are  shared 
between  collaborating  vendors.  However,  software  developed  by  competing  commer¬ 
cial  vendors  is  not.  IP  rights  to  LTE  software  are  retained  by  individual  vendors  in 
most  cases.  In  comparison,  JTRS  GMR  WNW  architecture  is  software-defined:  The 
software  is  government-owned  and  can  be  shared  between  vendors,  if  approved  by  the 
government,  whereas  GMR  hardware  is  not  shared  outside  the  original  development 
team  of  contractors. 

The  organizational  model,  and  technology-and  intellectual  property  sharing  rules 
differ  substantially  in  the  two  approaches.  The  JTRS  GMR  and  WNW  products  were 
developed  in  traditional  DoD  acquisition  programs  with  IP  access  controlled  by  a 
select  number  of  program  contractors  and  government  personnel.  IP  for  executable 
software  code  is  retained  by  the  government,  whereas  individual  contractors  retain  IP 
for  hardware  designs.  GMR  and  WNW  were  developed  by  two  acquisition  programs, 
one  responsible  predominantly  for  hardware  development  and  for  the  operating  system 
(OS)  software  for  the  hardware  platform  (the  GMR  program),  and  the  other  (the  JTRS 
NED  program)  responsible  for  WNW  (a  software  product).  One  complication  of  this 
arrangement  is  that  a  significant  amount  of  coordination  was  required  between  the 
two  programs  for  software  development.  The  JTRS  Joint  Program  Executive  Office 
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(JPEO)  was  responsible  for  coordinating  and  aligning  the  products  of  the  GMR  and 
NED  programs.  One  mechanism  used  to  effect  the  coordination  of  software  develop¬ 
ment  efforts  was  the  development  of  a  set  of  software  standards  for  the  JTRS  family  of 
radios — the  software  communications  architecture  (SCA). 

In  comparison,  the  LTE/system  architecture  evolution  (SAE)  is  specified  by  a  set 
of  technical  standards  that  was  developed  by  the  3GPP  international  consortium.  The 
consortium’s  governing  body  is  a  global  set  of  regional  telecommunications  standards- 
setting  organizations.  3GPP  also  includes  industry  firms.  LTE/SAE  standards  are 
developed  by  industry  experts  from  member  firms.  Individual  firms  have  the  option  to 
offer  their  own  IP  to  form  a  part  of  one  or  more  LTE/SAE  standards.  If  accepted  by  the 
group,  this  IP  is  then  available  to  other  3GPP  members  to  use  in  their  own  products 
under  the  terms  of  the  3GPP  essential  patent  licensing  policy.  Intellectual  property 
rights  (IPR)  are  retained  by  the  individual  3GPP  member  firms,  but  licenses  for  essen¬ 
tial  LTE/SAE  patents  must  be  made  available  to  other  3GPP  member  firms  on  a  fair, 
reasonable,  and  nondiscriminatory  basis. 

LTE  Had  Much  Higher  Development  Costs  Than  JRTS  GMR  WNW 

One  of  the  most  surprising  results  of  this  study  concerns  RDT&E  costs.  Significantly 
more  funds  were  expended  for  the  development  of  LTE  than  was  allocated  by  the  DoD 
for  the  development  of  JTRS  GMR  WNW.  We  note  that  some  of  the  difference  may 
have  stemmed  from  redundant  investments  by  industry  competitors,  but  we  tried  to 
take  into  account  this  effect  in  our  cost-analysis  methodology.  LTE  RDT&E  costs 
were  at  least  a  factor  of  four  greater  than  those  for  GMR  and  WNW  and  were  more 
likely  to  be  a  factor  of  10  to  24  times  greater. 

This  finding  raises  the  question  as  to  whether  the  DoD  did  not  make  a  large 
enough  research  and  development  (R&D)  investment  to  reduce  the  substantial  techni¬ 
cal  risks  associated  with  the  advanced  technologies  needed  for  this  program. 

Of  course,  JTRS  is  not  the  only  DoD  program  to  suffer  significant  cost  growth 
over  the  past  decade.  And  perhaps  what  has  been  observed  with  JTRS  is  just  symp¬ 
tomatic  of  a  larger  issue  within  the  DoD,  where  cost-estimation  processes  lead  to  overly 
optimistic  and  ultimately  unrealistic  cost  estimates  for  DoD  acquisition  programs  that 
require  the  development  of  advanced  technologies. 

JTRS  GMR  WNW  and  LTE  Development  Time  Lines  and  Approaches  Differ 
Substantially 

The  development  time  lines  overall  for  JTRS  GMR  WNW  and  for  LTE  are  compa¬ 
rable.  Initial  requirements  development  for  the  GMR  program  took  approximately  six 
years  compared  to  about  five  years  for  LTE.  On  the  other  hand,  system  development 
took  nine  years  for  JTRS  GMR  and  WNW  and  only  six  years  for  LTE.  One  reason 
for  the  significant  difference  in  the  latter  time  line  is  that  there  was  more  concurrent 
development  of  requirements  and  systems  in  the  LTE  program. 
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WNW  and  LTE  share  common  technologies  in  radio  frequency  modulation 
schemes,  but  LTE  incorporates  some  advanced  technologies  not  available  in  WNW 
such  as  multiple-input  and  multiple-output  (MIMO)  and  the  use  of  multiple  antennas 
in  handsets  and  eNode  B  radio  access  network  nodes.  LTE  also  employs  a  new,  power- 
efficient,  high-performance  uplink  waveform  and  wider  bandwidth  downlink  channels 
than  are  available  in  WNW  The  LTE  network  architecture  has  demonstrated  that  it  is 
scalable  and  can  support  over  1,000  nodes  in  a  single  wireless  network. 

LTE  and  WNW  network  and  security  architectures  are  substantially  different. 
JTRS  WNW  is  a  single-tier,  peer-to-peer  network  where  each  node  is  identical  and 
has  all  the  necessary  intelligence  built  into  it  so  that  it  can  function  effectively  even 
on  the  move  as  part  of  a  mobile  ad  hoc  network  (MANET).  As  held  tests  have  shown, 
WNW  network  scalability  has  not  been  demonstrated  beyond  30  nodes.  In  contrast, 
the  LTE/SAE  network  architecture  is  based  on  a  two-tier  network  where  a  single  radio 
access  point  (eNode  B)  can  support  over  1,000  wireless  nodes.  Finally,  both  network 
architectures  have  yet  to  fully  integrate  voice  and  data  communications. 

The  development  approaches  for  the  two  initiatives  also  differ  significantly. 
The  JTRS  program  approach  initially  called  for  the  development  of  an  entirely  new 
MANET  communications  architecture  that  relied  on  a  number  of  new  and  unproven 
technologies.  In  other  words,  it  used  a  big  bang  type  of  an  approach  where  it  was 
assumed  that  an  entirely  new  system  could  be  developed  and  that  the  new  technolo¬ 
gies  needed  to  make  the  system  work  could  be  developed  concurrently  with  the  system 
itself.  Later,  after  the  program  had  encountered  significant  cost,  performance,  and 
schedule  problems,  the  program’s  organizational  approach  was  changed  significantly. 
In  the  restructured  program,  hardware  and  software  development  was  separated  into 
separate  programs.  It  is  debatable  whether  this  new  organizational  approach  was  a 
success. 

The  LTE  development  approach  was  decidedly  different.  As  its  name  suggests, 
private  industry  and  3GPP  took  an  evolutionary  approach  toward  development  of  this 
new  system  architecture.  LTE  represents  an  incremental  development  approach  for 
next-generation  wireless  cellular  communications  networks.  Just  as  in  current  3G  wire¬ 
less  networks,  voice  and  data  backhaul  networks  are  separated  in  the  initial  versions 
of  LTE.2  It  was  judged  to  be  too  difficult  to  integrate  them  along  with  all  the  other 
architectural  changes  that  were  made  in  the  initial  versions  of  the  LTE  network  archi¬ 
tecture.  In  these  initial  versions,  development  emphasis  was  placed  on  improving  the 
spectral  efficiency  and  performance  of  the  radio  access  network,  on  consolidating  the 
number  of  network  nodes  needed  in  the  backhaul  network,  and  finally  on  transition¬ 
ing  the  backhaul  network  to  a  standard  IP-based  network.  And  perhaps  most  impor- 


2  The  cell  towers  of  wireless  mobile  communications  networks  are  connected  to  other  cell  towers  and  the  core 
network  of  the  mobile  communications  provider  by  means  of  a  backhaul  network.  The  backhaul  network  may 
refer  to  a  wire  line  or  fiber  optic  network,  or  high-capacity  wireless  links. 


Summary  xv 


tant,  the  fundamental  two-tiered  structure  of  current  wireless  cellular  networks  was 
preserved  in  LTE.  In  this  rwo-tiered  structure,  rhe  connection  between  the  two  LTE 
network  tiers  is  provided  by  the  eNode  B  network  node,  which  remains  fixed.  This  is 
in  contrast  to  the  MANET  network  designs  that  were  developed  as  part  of  the  JTRS 
GMR  and  WNW  programs,  in  which  no  network  nodes  have  to  remain  stationary. 
The  evolutionary  approach  taken  in  the  LTE  development  enabled  the  LTE  network 
to  scale  up  to  thousands  of  wireless  nodes;  in  contrast,  the  MANET  network  design 
developed  for  GMR  and  WNW  was  never  able  to  scale  up  to  such  large  network  sizes. 
The  technology  needed  to  do  this  remains  to  be  developed. 

Program  Outcomes  Were  Very  Different 

It  is  important  to  consider  the  outcomes  of  these  two  development  efforts.  The  JTRS 
GMR  WNW  development  effort  lasted  for  approximately  12  years  and  has  not  yet 
resulted  in  a  product  fielded  to  the  U.S.  Army.  The  GMR  program  was  canceled  in 
2011,  and  the  JTRS  program  responsible  for  the  development  of  WNW  was  restruc¬ 
tured  in  2012  and  is  now  called  the  Joint  Tactical  Networking  Center  (JTNC).  The 
JTNC  will  be  responsible  for  the  continued  development  of  JTRS  waveforms  and  will 
assume  the  responsibility  for  transitioning  these  waveforms,  including  WNW,  to  com¬ 
mercial  radios  (radios  developed  by  defense  contractors  using  their  own  internal  R&D 
funds).  So  in  terms  of  hardware  development  efforts,  the  JTRS  GMR  program  was 
not  a  success  as  it  did  not  lead  to  fielded  hardware  products.  In  contrast,  the  waveform 
development  part  of  the  JTRS  family  of  programs  (JTRS  NED)  was  more  success¬ 
ful  and  has  produced  advanced  networking  waveforms  that  will  be  used  in  radios  to 
fielded  to  U.S.  Army  and  Marine  Corps. 

Recommendations 

Our  analysis  shows  that  in  hindsight,  GMR  and  WNW  development  costs  do  not 
appear  to  be  exorbitant,  given  the  challenging  requirements  originally  established  for 
the  JTRS  program,  and  perhaps  the  DoD  did  not  invest  enough  in  up-front  R&D  to 
reduce  WNW  technical  risks.  If  WNW  is  to  be  scaled  up  to  larger  networks  of  100 
nodes  or  more,  additional  RDT&E  funds  will  be  necessary.  Such  an  R&D  program 
could  be  carried  out  by  JNTC  or  by  the  Army’s  Research,  Development  and  Engineer¬ 
ing  Command,  using  commercial  radios,  when  they  become  available. 

Another  approach  will  be  necessary  if  the  Army  and  DoD  decide  that  WNW 
cannot  meet  future  warfighter  needs.  If  this  is  the  case,  given  the  limited  R&D  dollars 
likely  to  be  available  in  future  budgets,  the  DoD  should  consider  adapting  LTE  to  meet 
its  operational  and  security  needs.  In  this  case,  a  new  DoD  R&D  investment  strategy 
will  be  needed  that  leverages  best  practices  from  the  commercial  sector,  including  the 
use  of  technical  standards,  to  foster  collaboration  among  competing  defense  contrac¬ 
tors.  Such  an  approach  for  the  DoD  would  be  new  but  could  take  advantage  of  the 
model  established  by  3GPR 
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In  any  case,  alternatives  to  the  JTRS  development  approach  based  on  software- 
defined  radios  and  waveforms  should  be  considered  in  future  DoD  communications 
systems  developments.  The  JTRS  SCA  is  aging  quickly  in  a  world  of  rapidly  changing 
software  architecture  standards  and  may  no  longer  provide  the  best  standards  plat¬ 
form  for  radio  development.  Furthermore,  it  is  not  consistent  with  recent  industry 
trends  toward  tighter  integration  of  software  and  hardware.  A  technical  standards- 
based  approach  that  also  incorporates  provisions  for  licensing  essential  patents  at  fair 
and  reasonable  prices  and  in  a  nondiscriminating  fashion  should  be  core  tenets  of  this 
new  R&D  strategy. 

In  addition,  alternatives  to  the  standard  DoD  acquisition  process  should  be  sought 
out  for  DoD  IT  and  communications  system  programs.  DoD  IT  and  communications 
should  follow  the  LTE  development  model  and  pursue  an  evolutionary  approach.  Such 
an  approach  is  possible  if  essential  patents  for  current  and  future  (evolved)  systems 
can  be  licensed  in  a  fair  and  non-discriminatory  basis  with  competing  contractors. 
DoD  should  also  initiate  such  programs  with  a  new  standards-development  phase  that 
enables  a  broad  cross-section  of  industry  to  contribute  and  share  patents  deemed  essen¬ 
tial  for  system  development.  DoD  should  also  consider  altering  later  program  phases 
to  enable  broader  industry  competition  that  such  an  IPR-sharing  approach  will  enable. 

Finally,  the  DoD  should  cast  a  wider  technology  net  to  incorporate  the  latest  com¬ 
mercially  developed  advanced  information  technologies.  To  do  this,  the  DoD  should 
consider  joining  3GPP  and  explore  the  option  of  enabling  3GPP  common  essential 
patents  to  be  used  in  military  system. 


Multiple  Breaches  Findings 

The  Performance  Assessments  and  Root  Cause  Analysis  (PARCA)  office  asked  RAND 
to  review  programs  that  had  breached  Nunn-McCurdy  thresholds  more  than  once  with 
the  main  objective  of  understanding  how  these  program  breaches  evolved.  PARCA 
posed  the  following  three  research  questions: 

•  Are  programs  that  have  a  Nunn-McCurdy  breach  more  likely  to  breach  again? 

•  What  can  we  learn  from  the  cost  growth  trends  of  those  programs  that  have  had 
multiple  Nunn-McCurdy  breaches? 

•  What  can  we  learn  from  the  management  actions  taken  on  programs  with  mul¬ 
tiple  breaches?  Are  there  common  issues  that  were  missed  at  the  first  breach? 

At  the  outset,  we  note  that  some  repeat  Nunn-McCurdy  breaches  can  be  attrib¬ 
uted  in  part  to  changes  in  legislation  rather  than  new  program  issues.  Furthermore, 
counting  breaches  is  not  as  simple  as  it  might  appear  on  the  surface.  A  Nunn-McCurdy 
breach  can  either  be  significant  or  critical,  depending  on  by  how  much  the  program 
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breaches  the  threshold.3  For  example,  does  a  transition  from  significant  breach  to  a 
critical  one  count  as  an  additional  breach?  We  took  as  simple  an  approach  as  possible. 
Once  a  program  experiences  either  an  average  procurement  unit  cost  (APUC)  or  a 
program  acquisition  unit  cost  (PAUC)  breach,  either  significant  or  critical,  that  consti¬ 
tutes  a  breach.4  We  did  not  consider  a  program  going  from  significant  to  critical  as  an 
additional  breach  but  rather  regarded  it  as  just  a  further  evolution  of  the  same  breach. 
Likewise,  we  observed  generally  that  APUC  and  PAUC  were  so  correlated  in  terms 
of  breaches  that  counting  breaches  of  APUC  and  PAUC  independently  did  not  make 
sense.  Also,  if  one  or  the  other  lags  by  a  year  or  so,  that  would  not  constitute  a  second 
breach.  For  example,  if  a  program  had  a  critical  breach  in  APUC  one  fiscal  year  and 
critical  or  significant  breach  in  PAUC  the  following  year,  we  did  not  count  that  as  two 
breaches  but  rather  as  a  single  breach.  The  only  time  we  restarted  the  counting  is  when 
there  was  a  rebaseline  (before  2006)  or  a  recertification.  We  considered  18  programs 
with  multiple  breaches,  four  in  depth. 

Our  broad  conclusions  are  as  follows: 

Are  Programs  with  Multiple  Breaches  More  Likely  to  Breach  Again? 

Our  analysis  indicates  that  programs  that  breach  once  are  not  more  likely  to  breach  a 
second  time.  The  2006  change  in  the  Nunn-McCurdy  law  has  increased  awareness  of 
programs  that  breach  above  their  initial  baselines. 

What  Can  We  Learn  from  the  Cost  Growth  Trends  of  Those  Programs  That  Have 
Had  Multiple  Nunn-Mccurdy  Breaches? 

We  found  no  obvious  cost  growth  trends  that  would  suggest  a  program  might  breach 
more  than  once. 

What  Can  We  Learn  from  the  Management  Actions  Taken  on  Programs  with 
Multiple  Breaches?  Are  There  Common  Issues  That  Were  Missed  at  the  First  Breach? 

In  terms  of  actions  taken  at  the  first  breach,  those  that  breach  more  than  once  had 
technical  issues  that  were  not  resolved  by  corrective  actions  taken  at  the  first  breach. 
Finally,  we  did  find  some  common  characteristics  among  those  programs  with  multiple 
breaches.  But  because  the  sample  was  small,  the  results  are  not  definitive,  and  further 
research  might  refine  this  view. 


3  A  significant  breach  occurs  if  the  program  exceeds  the  current  baseline  estimate  by  15  percent  or  the  original 
baseline  estimate  by  30  percent;  a  breach  is  critical  if  the  program  exceeds  the  current  or  original  baselines  by  25 
or  50  percent,  respectively. 

The  methods  for  calculating  the  breaches  are  explained  in  Appendix  A. 
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2G 
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3G 
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3GPP 
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AMP 

Avionics  Modernization  Program 

AoA 
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Amphibious  Ready  Group 

ASR 

acquisition  stability  reserve 

ATR 

above  threshold  reprogramming 
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authentication  center 

BCT 

brigade  combat  team 

BY 

base  year 

C4 

command,  control,  communications,  and  computers 

CAAP 

Common  Avionics  Architecture  Program 

CAIG 

Cost  Analysis  Improvement  Group 

Cdma 

Code  Division  Multiple  Access 

CMSP 

commercial  mobile  service  provider 

CORBA 

Common  Object  Request  Broker  Architecture 

COTS 

commercial  off  the  shelf 

DAB 

Defense  Acquisition  Board 

DACA 

deployable  aerial  communications  architecture 

DAE 

Defense  Acquisition  Executive 

DAMIR 

Defense  Acquisition  Management  Information  Retrieval 

DARPA 

Defense  Advanced  Research  Projects  Agency 

DCGS-IC 

Distributed  Common  Ground  Station-Intelligence  Community 

DISA 

Defense  Information  Systems  Agency 

DoD 

Department  of  Defense 

EAC 

estimate  at  completion 

EDM 

engineering  development  model 

EMD 

engineering  and  manufacturing  development 

EMU 

Engineering  Mockup  Unit 

ENB 

enhanced  Node  B 

EPC 

evolved  packet  core 

EPLRS 

Enhanced  Position  Location  Reporting  System 

EPS 

evolved  packet  system 
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European  Telecommunications  Standards  Institute 
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E-UTRA 

Evolved  UMTS  Terrestrial  Radio  Access 

E-UTRAN 

Evolved  UMTS  Terrestrial  Radio  Access  Network 

EVM 

earned  value  management 

EW 

electronic  warfare 

FCC 

Federal  Communications  Commission 

FCS 

Future  Combat  System 

FMV 

full  motion  video 

FUE 

first  unit  equipped 

FSS 

flight  software  system 

FY 

fiscal  year 

GAO 

Government  Accountability  Office 

GEO 

geosynchronous  earth  orbit 

GMR 

Ground  Mobile  Radio 

GPS 

Global  Positioning  System 

GPRS 

General  Packet  Radio  Service 

GSM 

Global  System  for  Mobile  Communications 

GW 

gateway 

HAIPE 

high- assurance  Internet  protocol  encryptor 

HEO 

highly  elliptical  orbit 

HF 

high  frequency 

HMS 

Handheld  Manpack  Small  Form  Fit 

HSDPA 

High-Speed  Downlink  Protocol  Access 

HSS 

home  subscriber  server 

HW 

hardware 

IA 

information  assurance 

ICE 

independent  cost  estimate 

INFOSEC 

information  security 
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IOC 

initial  operating  capability 

IP 

intellectual  property;  Internet  protocol 

IPR 

intellectual  property  rights 

IT 

information  technology 

JCTD 

Joint  Capability  Technology  Demonstration 

JITC 

Joint  Interoperability  Test  Command 

JPATS 

Joint  Primary  Aircraft  Training  System 

JPEO 

Joint  Program  Executive  Office 

JROC 

Joint  Requirements  Oversight  Council 

JSOC 

Joint  Special  Operations  Command 

JTN 

Joint  Tactical  Network 

JTNC 

Joint  Tactical  Networking  Center 

JTRS 

Joint  Tactical  Radio  System 

KPP 

key  performance  parameter 

L1/L2 

Layer  1 /Layer  2 

LRIP 

low-rate  initial  production 

LSTI 

LTE/SAE  Trial  Initiative 

LTE 

long-term  evolution 

MANET 

mobile  ad  hoc  network 

MBITR 

Multiband  Inter/Intra  Team  Radio 

MCS 

Mission  Control  Station 

MDAP 

major  defense  acquisition  program 

MHAL 

modem  hardware  abstraction  layer 

MILS 

multiple  independent  levels  of  security 

MIMO 

multiple-input  and  multiple- output 

MME 

mobility  management  entity 

NAS 

nonaccess  stratum 
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NAVAIR 

Naval  Air  Systems  Command 

NDAA 

National  Defense  Authorization  Act 

NED 

network  enterprise  domain 

NIU 

network  INFOSEC  unit 

NS  A 

National  Security  Agency 

NTDR 

Near  Term  Digital  Radio 

OEM 

original  equipment  manufacturer 

OFDM 

orthogonal  frequency-division  multiplexing 

OF  DMA 

orthogonal  frequency-division  multiplex  access 

OPEVAL 

Operational  Evaluation 

ORB 

object  request  broker 

ORD 

Operational  Requirements  Document 

OS 

operating  system 

OSD 

Office  of  the  Secretary  of  Defense 

PARCA 

Performance  Assessments  and  Root  Cause  Analysis 

PAUC 

program  acquisition  unit  cost 

PB 

President’s  budget 

PC 

personal  computer 

PCRF 

policy  and  charging  rules  function 

PDN 

packet  data  network 

PDR 

Program  Deviation  Report 

PEO 

program  executive  office 

PMB 

Program  Management  Board 

QAM 

quadrature  amplitude  modulation 

QoS 

quality  of  service 

QPSK 

quadrature  phase-shift  keying 

R&D 

research  and  development 
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RAN 

radio  access  network 

RBS 

radio  base  station 

RDT&E 

research,  development,  test,  and  evaluation 

RFP 

request  for  proposal 

RRC 

radio  resource  control 

RTOS 

real-time  operating  system 

SAE 

system  architecture  evolution 

SAR 

Selected  Acquisition  Report 

SATCOM 

satellite  communications 

SBIRS 

Space-Based  Infrared  System 

SBU 

sensitive  but  unclassified 

SCA 

software  communications  architecture 

SC-FDMA 

single-carrier  frequency  division  multiple  access 

SCP 

service  cost  position 

SDD 

system  development  and  demonstration 

SDO 

Standards  Development  Organization 

SDR 

software-defined  radio 

SEC 

Securities  and  Exchange  Commission 

SECDEF 

Secretary  of  Defense 

SFR 

System  Functional  Review 

SGW 

services  gateway 

SINCGARS 

Single  Channel  Ground  and  Airborne  Radio  System 

SIT 

system  integration  test 

SRW 

soldier  radio  waveform 

SSN 

nuclear  submarine 

SWE/NOR 

Sweden/Norway 

TACP 

Tactical  Air  Control  Party 
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TS 

TSPR 

UAS 

UE 

UHF 
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UMTS 

USD  (AT&L) 

VHF 

VoIP 

WiMAX 
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WSARA 


transmission  control  protocol 

training  information  management  system 

top  secret 

Total  System  Performance  Responsibility 
unmanned  aircraft  system 
user  equipment 
ultra  high  frequency 
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Universal  Mobile  Telecommunications  Standard 

Under  Secretary  of  Defense  (Acquisition,  Technology,  and  Fogistics) 

very  high  frequency 

voice  over  Internet  protocol 

Worldwide  Interoperability  for  Microwave  Access 

wideband  networking  waveform 

Weapon  System  Acquisition  Reform  Act 


CHAPTER  ONE 


Joint  Tactical  Radio  System 


This  chapter  presents  the  results  of  the  study  examining  the  Army’s  JTRS  program  that 
uses  industry’s  LTE  waveform.  It  begins  with  a  discussion  of  the  JTRS  program  back¬ 
ground  and  then  describes  the  objectives  of  the  research.  Next,  we  discuss  the  analytic 
approach  taken  to  accomplish  the  objectives.  The  following  section  describes  the  JTRS 
architecture,  to  include  a  discussion  of  the  JTRS  hardware,  software,  and  security 
architecture.  We  then  summarize  the  LTE  architecture,  to  include  a  discussion  of  the 
LTE  security  architecture,  software  architecture,  and  description  of  commercial  and 
DoD  deployments.  Then  we  compare  the  development  approaches  and  organizational 
structures  for  JTRS  and  LTE,  and  this  discussion  is  followed  by  a  technical  perfor¬ 
mance  comparison,  to  include  a  comparison  of  the  development  time  lines  associated 
with  both  systems.  We  then  focus  on  development  costs  for  both  systems,  to  include  a 
discussion  of  the  sources  and  methods  used  to  calculate  the  associated  costs.  Finally,  we 
discuss  the  findings  and  implications  derived  from  the  analysis  in  this  report. 


Background  of  the  JTRS  Program 

On  the  surface,  the  DoD’s  software  development  efforts  in  support  of  the  JTRS  appear 
to  have  been  costly  and  time-consuming  and  to  have  led  to  the  development  of  sys¬ 
tems  that  do  not  meet  their  original  operational  requirements.  Such  factors  have  con¬ 
tributed  to  a  Nunn-McCurdy  breach  for  the  flagship  program  of  the  JTRS  family  of 
programs — the  Ground  Mobile  Radio  (GMR),  and  later  its  cancelation.  RAND  inde¬ 
pendently  assessed  the  JTRS  GMR  program  and,  used  the  results  of  this  assessment 
in  this  analysis.1 

JTRS  is  a  family  of  radios  and  a  family  of  communications  waveforms  designed 
to  be  interoperable  and  provide  U.S.  military  forces  with  next-generation  systems  for 
digital  voice  and  data  communications  during  military  operations.  Interoperability 
for  JTRS  has  two  dimensions.  First,  two  JTRS  radios  using  the  same  waveform  are 


1  The  results  of  the  RAND  independent  assessment  of  the  JTRS  GMR  program  are  not  available  to  the  general 
public. 
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designed  to  exchange  information  effectively.  Second,  JTRS  software-defined  wave¬ 
forms  developed  originally  on  one  JTRS  hardware  platform  or  radio  are  designed  to 
be  portable  and  to  function  effectively  on  a  different  JTRS  radio.  The  transfer  of  a 
waveform  from  one  hardware  platform  to  another  is  called  waveform  porting.  JTRS 
waveform  porting  is  theoretically  made  possible  through  the  use  of  a  software  com¬ 
munications  architecture  (SCA)  that  provides  a  common  software  “operating  system” 
(OS)  for  JTRS  radios,  much  like  Microsoft  Windows  provides  a  common  operating 
system  for  personal  computers  developed  by  different  manufacturers. 

The  above  discussion  indicates  that  key  software  development  items  are  JTRS 
waveforms.  A  waveform  provides  the  key  wireless  communications  functionality  that 
defines  how  information  is  encoded  on  the  radio  frequency  emissions  of  the  radio. 
But  in  the  JTRS  program,  waveform  has  a  broader  meaning  and  definition  because 
it  includes  the  entire  set  of  protocols  and  standards  that  are  used  to  manage  the  wire¬ 
less  network  and  to  encode  information  as  Internet  protocol  (IP)  packets  used  by  the 
radio  itself.  During  the  development  of  the  JTRS  radios,  in  particular  the  develop¬ 
ment  of  the  GMR  and  the  wideband  networking  waveform  (WNW) — the  most  ambi¬ 
tious  and  complex  waveform  developed  in  the  entire  program — it  became  clear  that 
development  of  these  key  software  components  took  longer  and  was  more  difficult 
than  originally  anticipated.  This  led  to  schedule  delays  that  affected  the  JTRS  GMR 
program. 


Objectives  of  the  Research 

The  history  of  the  JTRS  program  raises  several  questions:  blow  do  commercial  indus¬ 
try  development  processes  differ  from  those  used  in  the  JTRS  program?  Does  com¬ 
mercial  industry  produce  radios  and  associated  software  more  effectively  and  efficiently 
than  the  DoD?  Can  the  DoD  learn  from  commercial  industry  better  ways  to  develop 
wireless  communications  systems  that  could  be  applied  to  future  DoD  programs? 

The  objective  of  our  effort  is  to  determine  whether  DoD  software  development 
and  related  acquisition  processes  are  more  costly  and  time-consuming  than  com¬ 
mercial  approaches.  We  compare  the  development  approaches  for  the  JTRS  WNW 
to  those  of  a  similar  commercial  waveform.  The  commercial  waveform  selected  for 
comparison  was  the  4G  cellular  telephone  communications  system  that  uses  the  LTE 
waveform.  We  selected  LTE  because  it  is  the  most  advanced  waveform  developed  to 
date  by  commercial  industry  for  wireless  cellular  communications  and  is  being  widely 
adopted  around  the  world  for  next-generation  4G  cellular  networks.  JTRS  WNW  is 
the  most  advanced  wireless  communication  waveform  developed  by  the  DoD  for  ter¬ 
restrial  communications. 
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Analytical  Approach 

As  part  of  our  effort,  we  compare  WNW  and  LTE  performance  requirements,  their 
associated  system  architectures,  development  time  lines,  technical  complexity,  tech¬ 
nical  risks,  organizational  structures,  and  associated  costs.  Given  that  LTE  network 
nodes  (radio  terminals  and  handsets)  are  not  software-defined  and  that  vendor- 
specific  implementations  of  the  LTE  waveforms  are  not  hardware-independent,  this 
effort  compares  associated  hardware  and  software  research  and  development  (R&D) 
costs  and  capabilities  for  both  LTE  and  WNW  (e.g.,  the  JTRS  GMR). 

Originally,  we  focused  only  on  comparing  the  cost  of  developing  the  waveforms. 
However,  as  we  analyzed  the  LTE  development  process  and  system  architecture,  we 
determined  that  it  did  not  make  sense  to  compare  only  the  development  process  for 
waveforms  because  of  the  tight  integration  between  hardware  and  software  that  char¬ 
acterizes  the  commercial  radio  development  process.  For  this  reason,  we  decided  to 
include  the  hardware  development  process  as  well  as  the  software  development  process 
in  our  analysis. 

Development  approaches  for  DoD  systems  and  commercial  systems  differ  in  sev¬ 
eral  ways.  For  JTRS,  it  is  easy  to  separate  the  hardware  from  the  software  in  the  devel¬ 
opment  model.  In  fact,  the  development  model  is  designed  to  enable  a  complete  separa¬ 
tion  of  those  products  so  that  different  vendors  can  deliver  them. 

The  commercial  world  takes  a  completely  different  approach.  Its  approach  is  based 
on  establishing  standards  for  key  functionality  of  the  wireless  communications  device 
and  network.  It  also  has  been  structured  to  enable  competing  firms  to  offer  intellectual 
property  and  patents  to  a  larger  consortium  of  developers.  The  consortium  can  then 
decide  how  intellectual  property  is  incorporated  into  the  standards  that  define  the 
wireless  communication  system.  The  product  of  this  cooperative  development  process 
is  not  hardware  or  software  but  rather  standards. 

Previous  RAND  studies  that  examined  the  JTRS  program  in  great  detail 
informed  our  analysis  of  the  JTRS  program.  RAND  has  been  active  in  the  analysis 
of  the  technical  capabilities  and  limitations  of  JTRS  radios  for  the  Assistant  Secretary 
of  Defense  for  Research  and  Engineering  and  also  for  the  U.S.  Army.  In  these  efforts, 
RAND  examined  how  JTRS  radios  can  be  used  to  build  large  complex  networks  to 
support  Army  brigades.2  We  relied  on  that  information  and  our  knowledge  of  the 
JTRS  program  to  develop  a  description  of  how  the  JTRS  family  of  programs  under 
the  JTRS  Joint  Program  Executive  Office  (JPEO)  developed  waveforms  and  associated 
hardware.  We  also  used  DoD  Selected  Acquisition  Reports  (SARs)  to  estimate  the  cost 
of  development  of  JTRS  hardware  and  software. 

A  separate  and  different  analytical  approach  was  necessary  to  analyze  LTE.  Sev¬ 
eral  resources  are  accessible  on  the  Internet  regarding  the  LTE  development  process 


2  The  results  of  these  RAND  analyses  are  not  available  to  the  general  public. 
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because  it  is  conducted  by  an  international  consortium,  the  Third-Generation  Partner¬ 
ship  Project  (3GPP).3  In  addition,  we  took  advantage  of  contacts  RAND  had  estab¬ 
lished  with  key  LTE  industry  firms.  We  used  these  contacts  to  identify  knowledgeable 
engineers  in  LTE  firms  who  were  familiar  with  the  LTE  development  process  and  with 
the  3GPP  consortium.  We  interviewed  several  individuals  in  industry  who  have  played 
a  key  role  in  the  development  of  the  LTE  waveform.  During  these  interviews,  we  were 
able  to  corroborate,  understand,  and  establish  a  more  detailed  description  of  the  devel¬ 
opment  processes  for  the  LTE  waveform.  We  provide  details  below  on  the  organiza¬ 
tional  structure  of  both  the  JTRS  program  and  LTE. 

The  analytical  approach  used  for  the  JTRS  cost  analysis  consisted  of  combin¬ 
ing  DoD  research,  development,  test,  and  evaluation  (RDT&E)  cost  estimates  for 
software  (WNW/soldier  radio  wavefore  [SRW])  and  hardware  (GMR).  Although 
ideal  cost  comparisons  would  isolate  hardware  and  software  development  and  pro¬ 
gram  management  costs,  the  original  iteration  of  the  JTRS  program  integrated  both 
the  hardware  and  waveforms  into  one  program  to  facilitate  management  of  program 
interdependencies.  Available  data  provide  only  a  rough  cost  division  of  both  efforts. 
Further,  the  restructure  in  2006  that  separated  both  programs  possibly  shifted  faults 
of  WNW  development  into  GMR  costs  through  unit  reductions.  Thus,  the  most  accu¬ 
rate  cost  for  the  development  of  the  JTRS  WNW  capability  is  obtained  by  combining 
the  software  and  hardware  estimates. 

The  cost  analysis  approach  for  LTE  was  significantly  different  because  this  type  of 
information  is  not  readily  accessible  because  of  the  number  of  U.S.  and  international 
firms  that  have  been  involved  in  the  development  process.  Because  of  uncertainties 
regarding  some  of  the  data  available,  a  number  of  different  cost  analysis  approaches 
were  used  to  develop  a  range  of  cost  estimates  for  LTE.  The  details  of  these  different 
cost  analysis  approaches  are  described  later  in  this  report. 

We  used  publicly  available  information  from  3GPP,  including  their  published 
time  lines  for  the  development  of  LTE  and  related  networking  standards,  and  inter¬ 
views  with  engineers  at  key  firms  that  participated  in  the  development  of  LTE  standards 
and  in  the  development  and  manufacturing  of  LTE-related  equipment.  We  also  made 
use  of  data  that  can  be  found  in  LTE  standards  and  related  performance  requirements. 

From  the  start  of  this  research,  we  were  aware  that  the  LTE  and  WNW  waveforms 
were  developed  to  support  different  mobile  network  operational  goals.  In  this  study,  we 
do  not  wish  to  leave  the  impression  that  these  two  systems  are  equivalent.  They  are  not. 
They  share  some  common  underlying  technologies  and  have  some  common  attributes, 
but  the  two  architectures  were  developed  for  two  very  different  sets  of  operational 
requirements,  or  in  the  parlance  used  in  the  commercial  world,  two  very  different  busi¬ 
ness  cases.  Below,  we  describe  these  architectures  and  the  differences  in  operational 
requirements.  Despite  these  differences,  it  is  instructive  to  examine  whether  the  two 


3  See  3GPP,  “The  Mobile  Broadband  Standard,”  website,  undated. 
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architectures  were  developed  using  similar  or  different  acquisition  processes  and  the 
cost  of  these  developments  using  these  acquisition  approaches.  The  majority  of  this 
report  is  devoted  to  these  questions. 

A  separate  question,  which  we  entertain  briefly  at  the  conclusion  of  this  report,  is 
whether  the  LTE  architecture  and  some  of  its  essential  technological  components  can 
be  adapted  to  provide  mobile  communications  to  military  tactical  users.  Such  an  adap¬ 
tation  may  require  significant  changes  to  LTE  equipment  and  architecture.  Although 
the  full  answer  to  this  question  is  beyond  the  scope  of  the  current  effort,  we  believe  that 
it  may  be  possible  to  satisfy  a  high  proportion  of  GMR  and  WNW  requirements  but 
certainly  not  all  of  the  operational  requirements  originally  levied  on  the  JTRS  develop¬ 
ment  programs. 


Analytical  Limitations 

Our  analysis  was  subject  to  certain  limitations  because  it  was  conducted  at  an  unclas¬ 
sified  level  and  did  not  include  a  full  examination  of  all  of  the  features  of  proprietary 
LTE  waveform  implementations.  Vendor-specific  implementations  of  LTE  may  not 
perform  at  the  maximum  levels  (e.g.,  in  terms  of  data  rates)  specified  in  the  3GPP 
standard.  Only  test  results  could  reveal  such  flaws,  and  such  test  data  were  not  exam¬ 
ined  in  this  study,  ffowever,  an  informal  review  of  published  commercial  carrier  LTE 
test  results  reveals  that  vendors  are  able  to  produce  equipment  that  can  meet  the  full 
capabilities  of  the  LTE  waveform  specified  in  Releases  8  and  9  of  the  LTE  standard. 
In  addition,  because  of  the  limitations  mentioned  above,  a  detailed  comparison  of  the 
security  features  of  the  two  different  networking  systems  was  not  possible.  We  also 
did  not  attempt  to  identify  all  possible  vulnerabilities  associated  with  the  two  network 
architectures.  Such  an  investigation  is  beyond  the  scope  of  the  current  study.  How¬ 
ever,  our  experience  with  several  other  commercial  and  DoD  communication  systems 
leads  us  to  feel  that  it  is  plausible  to  assume  that  the  LTE  architecture  contains  more 
vulnerabilities  and  has  more  security  issues  associated  with  it  than  GMR  and  WNW. 
Although  GMR  never  received  a  full  and  complete  information  assurance  (IA)  accredi¬ 
tation  by  the  National  Security  Agency  (NSA),  the  JTRS  program  office  took  many  of 
the  steps  needed  to  obtain  such  accreditation. 

In  addition,  it  is  well  known  that  the  LTE  waveform  is  not  designed  to  be  jam 
resistant.  It  is  designed  to  operate  in  a  benign  environment  where  other  users  of  the 
frequency  spectrum  will  not  violate  the  Federal  Communications  Commission  (FCC) 
spectrum  allocation  rules.  On  the  other  hand,  GMR  and  WNW  have  the  provision  to 
operate  in  a  contested  electronic  warfare  (EW)  environment.  These  additional  military 
features  are  unique  to  GMR  and  WNW.  Full  analysis  of  these  issues  would  require 
a  classified  study;  however,  we  can  surmise  at  the  unclassified  level  that  GMR  and 
WNW  do  have  additional  capabilities  that  are  not  present  in  the  LTE  waveform. 
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It  is  relatively  straightforward  to  estimate  the  R&D  costs  for  GMR  and  WNW 
developments.  However,  as  described  above,  we  found  it  more  difficult  to  estimate  the 
costs  of  R&D  activities  for  LTE,  because  R&D  budgets  for  particular  products  are 
typically  treated  as  proprietary  information  inside  private  brms.  However,  publicly 
listed  brms  do  provide  an  exact  accounting  of  their  total  R&D  spending  in  their  cor¬ 
porate  bscal  year.  We  used  this  information  to  estimate  LTE  R&D  costs.  We  also  had 
to  use  other  data,  including  patent  information,  to  provide  such  estimates.  The  cost 
analysis  section  of  this  report  provides  a  detailed  description  of  all  the  parameters  used 
in  developing  these  cost  estimates.  We  wish  to  point  out  that  because  we  did  not  have 
direct  access  to  detailed  corporate  bnancial  records,  there  are  some  limitations  in  our 
cost  analysis.  Because  of  this,  we  use  several  different  analytical  techniques  to  estimate 
these  costs. 


Architecture 

The  original  concept  for  the  JTRS  program  dates  to  1997  and  was  born  out  of  the 
need  to  make  DoD  radios  interoperable.  Earlier  DoD  radios,  especially  those  used  by 
the  different  military  services,  were  not  interoperable  except  in  very  limited  ways.  To 
address  this  shortcoming,  planners  and  technology  experts  in  the  DoD  had  envisioned 
an  entirely  new  development  approach  to  deliver  interoperable  radios  that  could  be 
used  by  all  military  services.  Because  JTRS  was  to  incorporate  the  wireless  communi¬ 
cations  requirements  of  all  four  services,  it  took  some  time  to  establish  joint  require¬ 
ments  for  the  program.  In  2001,  JTRS  requirements  were  established  and  the  program 
began  with  a  Milestone  B  decision  in  2002.  However,  the  program  development  was 
not  smooth  and  the  program  was  restructured  several  times  during  the  brst  bve  years  of 
development.  Initially,  the  program  began  as  bve  separate  radio  programs  (termed  the 
JTRS  Cluster  programs)  that  would  be  managed  by  a  single  program  executive  office 
(PEO).  This  program  structure  was  later  changed,  and  JTRS  hardware  and  software 
development  activities  were  placed  into  separate  programs.  We  will  describe  below  how 
this  restructuring  occurred  and  how  the  program  offices  were  changed. 

We  will  also  discuss  the  JTRS  architecture  as  it  evolved  from  the  beginning  of 
the  program  and  as  it  was  later  debned  in  the  2008-2009  time  frame  after  the  major 
restructuring  referred  to  above  was  completed.  In  this  study,  we  will  focus  on  only  a 
few  key  parts  of  the  overall  JTRS  family  of  programs:  GMR  and  WNW. 

The  GMR  program  was  responsible  for  the  development  of  the  hardware  required 
for  the  GMR  device.  The  JTRS  NED  program  was  responsible  for  all  waveforms  devel¬ 
oped  in  the  JTRS  family  of  programs  and,  in  particular,  for  the  waveforms  that  would 
run  on  GMR.  The  most  complex  and  capable  waveform  in  the  JTRS  family  is  WNW. 
Because  of  its  complexity,  WNW  requires  substantial  hardware  resources  (processing 
power  and  memory)  to  run  effectively.  When  it  was  originally  developed,  only  the 


Joint  Tactical  Radio  System  7 


GMR  prototype  radio  could  run  WNW.  Together,  GMR  and  WNW  can  substantiate 
aJTRS  WNW  network. 

JTRS  WNW  provides  a  peer-to-peer,  wireless  MANET  where  all  nodes  are  the 
same  and  no  master  nodes  or  base  stations  are  required.  The  lack  of  a  master  node  in 
the  network  is  desired  because  even  if  one  or  more  nodes  on  the  battlefield  are  dam¬ 
aged  or  destroyed,  the  network  will  continue  to  operate  effectively. 

Figure  1.1  illustrates  aJTRS  WNW  network  with  several  subnets  that  operate  at 
different  frequencies.  WNW  is  designed  to  provide  wireless  digital  IP-based  communi¬ 
cations.  Although  no  firm  requirement  was  ever  established  for  the  required  minimum 
size  of  the  WNW  network,  the  WNW  MANET  was  a  key  part  of  the  network  in 
the  Army’s  original  plans  for  the  Future  Combat  System  (FCS).  The  FCS  force  design 
relied  on  relatively  light  armored  vehicles,  new  high-technology  sensors,  communi¬ 
cations  systems,  and  weapons.  FCS  vehicles  were  designed  to  be  light,  so  that  they 
could  be  airlifted  by  C-130  transport  aircraft.  FCS-equipped  units  would  be  surviv- 
able  because  they  were  envisioned  to  be  capable  of  finding  and  striking  enemy  targets 
faster  and  from  longer  ranges  than  adversary  forces  could.  A  robust  communications 
network  was  necessary  to  enable  this  bold  new  “net  centric”  operational  concept.  For 
these  reasons,  the  FCS  and  JTRS  programs  were  linked.  FCS  force  designers  assumed 

Figure  1.1 
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that  a  large  number  of  JTRS  GMR  equipped  nodes,  running  WNW,  would  be  linked 
into  a  high-capacity,  low-latency  network  in  an  FCS-equipped  brigade.  This  network 
would  include  JTRS  nodes  running  WNW  and  SRW  and  a  smaller  number  of  combat 
vehicles  equipped  with  satellite  communications  capability.  In  other  words,  WNW 
was  a  key  ingredient  in  a  network  of  networks  that  was  to  include  over  1,500  nodes. 
As  stated  by  the  GAO,  “For  example,  current  plans  call  for  the  network  supporting  a 
BCT  to  include  more  than  5,000  nodes  on  over  1,500  radio  sets  running  at  least  four 
different  advanced  networking  waveforms.  .  .  .”4 

Later,  after  the  FCS  program  was  canceled,  the  Army  developed  plans  to  use 
GMR  and  WNW  as  a  core  part  of  the  tactical  network  of  Army  maneuver  brigade 
combat  teams  (BCTs).  These  brigades  were  envisioned  to  have  over  a  100  or  so  WNW- 
capable  nodes,  a  number  that  corresponded  to  the  number  of  key  leader  or  command 
positions  in  a  maneuver  BCT. 

JTRS  WNW  networks  were  designed  to  remain  robust  even  when  a  large  frac¬ 
tion  of  these  vehicles  were  on  the  move.  This  capability  was  needed  so  that  the  net¬ 
work  would  remain  functional  even  when  JTRS-equipped  vehicles  moved  over  dif¬ 
ficult  terrain.  In  this  case,  WNW  would  adapt  to  the  changing  network  topology  or 
connectivity  of  the  WNW  network  without  suffering  a  degradation  in  network  perfor¬ 
mance.  GMR  nodes  would  still  be  capable  of  transmitting  information  across  the  net¬ 
work  regardless  of  how  frequently  the  network  topology  changed  because  line-of-sight 
radio  connectivity  between  vehicles  was  obscured.  Another  key  aspect  of  the  network 
design  is  that  the  network  would  remain  effective  even  as  the  number  of  vehicles  or 
radio  nodes  in  the  network  increased.  This  latter  capability  is  called  moble  ad  hoc  net¬ 
work  (MANET)  scalability.  However,  as  mentioned  above,  no  firm  requirement  for 
MANET  scalability  was  established  in  the  original  JTRS  Operational  Requirements 
Document  (ORD). 

It  is  important  to  note  that  most  “legacy”  tactical  radios  used  by  the  Army  and 
Marine  Corps  over  the  last  decade  provide  data  communications  networks  that  are 
relatively  static  in  terms  of  their  network  design  or  topology.  Such  legacy  tactical  radios 
also  have  a  limit  as  to  how  many  radios  or  nodes  can  participate  in  a  single  wireless 
network.  What  was  new  and  more  demanding  technically  that  WNW  was  to  provide 
is  network  scalability  and  the  ability  of  the  network  to  adapt  and  change  as  network 
connectivity  changes  and  simultaneously  to  provide  the  same  network  capacity  or  data 
rates  to  individual  GMR  radio  nodes.  It  turns  out  that  providing  these  new  MANET 
capabilities  has  proven  much  more  difficult  to  do  than  originally  envisioned. 


4  Government  Accountability  Office,  “U.S.  GAO-Defense  Acquisitions:  Significant  Challenges  Ahead  in 
Developing  and  Demonstrating  Future  Combat  System’s  Network  and  Software,”  Washington,  D.C.,  2008. 
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JTRS  Hardware 

Another  key  requirement  for  GMR  that  proved  to  be  more  difficult  to  achieve  than 
originally  envisioned  was  to  provide  a  radio  that  could  fit  within  the  limited  space 
available  in  many  legacy  combat  vehicles  and  that  could  be  powered  by  the  power  sys¬ 
tems  of  the  vehicles. 

One  reason  why  the  GMR  radio  is  so  large  relative  to  legacy  tactical  radios  is 
that  it  would  provide  four  independent  communication  channels  with  one  of  these 
channels  operating  at  high  power  (100  W).  This  capability  requires  high-power  ampli¬ 
fiers  to  support  both  WNW  and  the  SRW.  SRW  is  a  waveform  that  shares  many  of 
the  same  characteristics  as  WNW  but  is  not  designed  to  provide  as  high  a  capacity 
as  WNW  or  operate  at  the  same  high-power  levels.  SRW  is  designed  to  operate  on 
smaller  radios,  including  handheld  radios  that  could  be  used  by  individual  soldiers  and 
at  shorter  ranges. 

The  final  proposed  version  of  GMR  hardware  is  shown  in  Figure  1.2.  Pre¬ 
engineering  development  model  (EDM)  GMR  platforms  are  significantly  less  capable. 
The  universal  transceivers  shown  in  the  figure  create  the  wireless  waveform  for  each 
channel.  The  network  INFOSEC  unit  (NIU)  also  shown  contains  an  NSA-approved 
encryption  device  that  encrypts  and  decrypts  communications  packets  for  each  chan¬ 
nel  and  that  can  also  route  packets  between  channels  and  to  the  external  interfaces 
of  the  radio,  which  can  be  connected  to  various  command  and  control  devices  in  the 
vehicle.  The  display  device  is  used  to  set  up  and  configure  GMR  and  WNW  networks. 


Figure  1.2 

JTRS  GMR  Hardware 


Network  INFOSEC  unit 


1 


*  k  k 


*  4 ,  t,  tt 


l  1}  b  G- 


I 


Universal 

transceivers 


Portable  control 
display  device 


RAND  MG7 171 15-1. 2 


10  DoD  and  Commercial  Advanced  Waveform  Developments  and  Programs  with  Nunn-McCurdy  Breaches 


One  additional  challenging  aspect  of  GMR  was  the  requirement  that  it  be  able 
to  operate  over  a  wide  spectrum  band  and  support  communication  channels  in  high 
frequency  (HF),  very  high  frequency  (VHF),  and  ultra  high  frequency  (UHF),  L-band 
(25  MFIz  to  2  GHz).  This  requires  including  a  number  of  different  power  amplifiers  in 
the  system  that  can  operate  in  these  different  bands  and  isolating  them  effectively  from 
one  another  to  prevent  electromagnetic  interference. 

JTRS  Security  Architecture 

A  key  aspect  of  the  JTRS  WNW  network  is  the  security  architecture  it  would  sup¬ 
port.  This  is  also  a  key  area  that  differs  significantly  from  what  would  be  available  in 
a  commercial  wireless  communication  system.  Interviews  with  program  officers  in  the 
JTRS  program  and  reviews  of  program  documentation  indicate  that  the  development 
of  the  JTRS  security  architecture  was  difficult  and  caused  some  program  delays  and 
may  have  been  the  source  of  some  program  cost  increases. 

WNW  security  architecture  supports  multiple  independent  levels  of  security 
(MILS).  This  enables  external  devices  and  data  sources  that  operate  at  different  secu¬ 
rity  levels  to  use  the  same  JTRS  WNW  network,  as  shown  in  Figure  1.3.  IP  packets 


Figure  1.3 

JTRS  WNW  Network  Security  Architecture 


Server  Workstation 


SOURCE:  Adapted  from  "Joint  Tactical  Radio  System  Update:  Tactical  Networking,"  Free  Online  Library, 
undated. 
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transmitted  between  nodes  in  the  WNW  network  would  be  encrypted  at  the  source 
node  and  decrypted  at  the  destination  node(s).  Nodes  that  relay  the  packets  between 
source  and  destination  would  not  decrypt  the  IP  packets.  This  makes  it  possible  to 
restrict  possession  of  the  cryptographic  keys  essential  to  reading  the  message  to  only 
the  recipient  and  transmitting  nodes.  Such  a  network  is  called  a  “black  core”  or  “color¬ 
less  core”  network  so  that  even  if  an  adversary  were  able  to  eavesdrop  on  a  WNW  net¬ 
work  and  extract  IP  packets  from  the  data  stream,  it  would  not  be  able  to  interpret  the 
message  unless  it  could  also  decrypt  the  packets.  The  encryption  device  that  enables 
such  a  black  core  network  to  be  formed  is  the  NSA- approved,  high-assurance  Internet 
protocol  packet  encryptor  (HAIPE). 

The  figure  shows  that  HAIPE  can  support  the  simultaneous  and  independent 
transmission  of  IP  packets  from  secret,  top  secret,  and  sensitive  but  unclassified  (SBU) 
network  enclaves.  Devices  in  these  network  enclaves  could  all  share  the  same  JTRS 
WNW  transport  layer.  This  is  made  possible  by  encrypting  information  at  the  packet 
level.  In  the  figure,  Black  IP  signifies  an  IP  network  in  which  IP  packet  payloads  are 
encrypted.  Only  the  packet  headers  remain  in  plain  text.  The  HAIPE  device  in  each 
GMR  can  decrypt  Black  IP  packets  with  the  encryption  key  appropriate  for  the  clas¬ 
sification  level  of  the  message  and  network  node  destination  and  convert  them  to  plain 
text  (Red  IP  packets).  The  advantage  of  this  type  of  security  architecture  is  that  it  can 
support  three  network  enclaves  operating  at  different  security  levels  with  only  one 
transport  network.  Absent  such  a  capability,  three  separate  transport  networks  would 
be  required  with  three  separate  encryption  devices. 

The  GMR  and  WNW  security  architecture  is  a  unique  capability  of  these  mili¬ 
tary  systems  and  provides  some  unique  advantages  for  securing  military  communica¬ 
tions.  Although  it  was  difficult  to  accomplish,  the  GMR  program  eventually  did  suc¬ 
ceed  in  obtaining  GMR  and  WNW  NSA  information  assurance  certification  for  these 
systems.  There  is  no  equivalent  capability  or  analog  available  in  the  LTE  architecture. 
The  latter  architecture  would  have  to  be  modified. 

JTRS  GMR  Software  Architecture 

As  mentioned  above,  after  the  JTRS  family  of  programs  was  restructured,  software 
and  hardware  development  activities  were  segregated  into  separate  programs.  GMR  is 
an  example  of  a  software-defined  radio  (SDR).  In  such  a  radio  system,  it  is  possible  to 
change  the  software  running  on  the  radio  without  changing  the  hardware.  In  this  way, 
it  is  theoretically  possible  to  upgrade  the  software  in  the  radio,  and  possibly  its  per¬ 
formance,  without  upgrading  the  hardware.  In  addition,  it  was  also  thought  that  this 
would  be  a  way  in  which  waveforms  developed  by  particular  vendors  could  be  loaded 
and  used  on  radios  developed  by  other  vendors.  In  other  words,  an  SDR  approach 
would  enable  more  competition  and  possibly  lower  cost  for  military  radios.  However, 
the  last  statement  must  be  considered  as  only  a  hypothesis  at  this  point  because  evi¬ 
dence  of  effective  competition  in  JTRS  radios  is  yet  to  emerge. 
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The  GMR  software  architecture  is  designed  to  conform  to  the  JTRS  software 
architecture,  which  all  JTRS  radios  comply  with.  The  basis  for  this  architecture  is  the 
SCA,  which  provides  a  framework  and  application  programming  interfaces  (APIs)  that 
enable  higher-level  software  components  (e.g.,  waveforms  such  as  WNW  and  SRW) 
to  be  separated  from  the  hardware  components  of  the  radio.  Shown  in  Figure  1.4  are 
some  of  the  key  software  components  of  the  JTRS  GMR  software  architecture. 

In  the  figure,  waveforms  are  indicated  by  the  brown  rectangles  at  the  top  of  the 
software  stack.  These  use  the  waveform  interface  and  other  radio  services  shown  in  the 
middleware  and  hardware  building  blocks  below.  The  software  components  shown  in 
green  constitute  the  middleware  and  operating  system  of  the  SDR.  GMR  was  based  on 
a  specialized  version  of  the  Linux  operating  system.  This  operating  system  was  specially 
configured  and  “locked  down”  to  conform  to  the  security  requirements  for  the  JTRS 
program  and  also  in  an  attempt  to  ensure  that  the  radio  could  perform  all  necessary 
information  processing  functions  in  real  time.  What  we  mean  by  this  is  that  informa¬ 
tion  that  was  to  be  transmitted  by  the  radio  would  be  ready  in  memory  for  retrieval  by 
the  waveform  so  that  it  could  be  transmitted  as  a  wireless  signal  at  the  required  data 
rate. 


Figure  1.4 

JTRS  GMR  and  WNW  Software 
Architecture 


\ 

WNW  I  SRW  I 

I  I 

NC- 

ARS 

SCA  core  framework 

r 

|  CORBA  ORB 

TCP/IP 

SCA  2.2 

L 

Secure  RTOS 

MHAL 

A/D  converters 

GIN 

/IR  hardware 

SOURCE:  Adapted  from  Douglas  C.  Schmidt, 
"Open  System  Architecture  (OSA):  Challenges 
&  Success  Drivers,"  Vanderbilt  University, 
Defense  News  Open  Architecture  Summit, 
October  18,  2012. 


RAND  MG117H5-1.4 


Joint  Tactical  Radio  System  13 


As  alluded  to  above,  another  important  requirement  for  the  software  architecture 
is  that  it  could  be  used  by  multiple  vendors  and  waveform  developers.  Consequently, 
the  top-level  APIs  used  in  the  architecture  are  defined  in  a  common  industry  format, 
which,  although  it  was  developed  for  other  purposes,  was  adapted  for  use  in  SDRs. 

Consequently,  the  SCA  core  framework  depends  on  the  real-time  object  request 
broker  (ORB)  of  the  CORBA  framework,  which  makes  systems  calls  to  the  underlying 
real-time  operating  system  (RTOS).  The  use  of  the  CORBA  framework  provides  an 
interoperability  framework  and  standardization  for  developers.  However,  its  use  in  the 
SCA  has  been  criticized  by  some  experts  because  it  relies  on  high-level  programming 
language  software  artifacts.  The  use  of  CORBA  in  a  radio  requires  complex  compil¬ 
ers  and  command  translation.  This  can  reduce  performance  and  increase  time  delays 
for  complex  CORBA  transactions.  In  this  regard,  it  should  be  noted  that  this  type  of 
software  development  approach  has  not  been  adopted  in  LTE.  Instead,  commercial 
developers  of  LTE  systems  and  components  use  their  own  proprietary  software  devel¬ 
opment  frameworks  and  maintain  intellectual  ownership  of  the  software  embedded  in 
their  systems. 

Another  challenge  in  the  development  of  the  GMR  and  WNW  software  archi¬ 
tecture  is  actually  connecting  it  efficiently  with  radio  hardware.  This  is  done  through 
two  types  of  components  shown  in  Figure  1.4:  the  modem  hardware  abstraction  layer 
(MHAL)  and  the  analog-to-digital  (A/D)  converters.  At  some  point,  waveform  data 
have  to  be  converted  into  analog  radio  signals.  This  is  done  by  the  MHAL  and  A/D 
converters.  And  the  reverse  has  to  be  done  for  incoming  waveform  radio  signals.  This 
is  also  done  by  the  same  baseband  components:  the  MHAL  and  A/D  converters.  This 
entire  architecture  requires  2.4  million  lines  of  software  code  in  addition  to  the  million 
lines  of  code  already  embedded  in  the  Linux  OS.5 


Long-Term  Evolution  Architecture 

Commercial  cell  phone  networks  have  evolved  rapidly  from  their  start  in  the  early 
1980s.  The  networks  have  been  upgraded  by  commercial  mobile  service  providers  in 
a  series  of  major  releases  or  network  generations.  The  latest  and  most  advanced  cell 
phone  network  is  referred  to  as  the  4G  network  or  LTE,  which  is  not  a  peer-to-peer 
network  as  is  WNW;  in  contrast,  it  has  two  scalable  network  tiers. 

eNode  B 

Users  with  wireless  mobile  devices  connect  to  the  radio  access  network  (RAN)  through 
enhanced  Node  Bs  (eNode  Bs)  wireless  network  access  points.  As  shown  in  Figure  1.5, 


5  A  complete  description  of  the  JTRS  GMR  and  WNW  software  architecture  is  beyond  the  scope  and  classifi¬ 
cation  of  this  report. 
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Figure  1.5 

LTE  Wireless  Cellular  Communications  Architecture 
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the  wireless  connection  between  user  equipment  (UE),  which  in  many  cases  will  be  cell 
phones,  and  the  eNode  B  is  indicated  by  the  dashed  lines  in  the  figure. 

Figure  1.5  shows  the  LTE  RAN  and  initial  services  gateway  (SGW)  entry  points 
into  the  core  LTE  communications  architecture.  The  central  node  of  the  RAN  is  the 
eNode  B,  which  is  typically  equipped  with  antennas  mounted  on  cell  towers  or  build¬ 
ings.  User  equipment  connects  to  the  eNode  B  using  the  wireless  channel  that  is  often 
referred  to  as  the  LTE  waveform  (we  describe  the  features  of  the  LTE  waveform  below). 
eNode  Bs  are  connected  into  the  LTE  backhaul  network  through  SGWs  by  means  of 
fiber  optic  links  or  high-capacity,  point-to-point,  wireless  links  as  shown  in  the  figure. 

The  LTE  waveform  provides  substantial  improvements  in  end-user  communica¬ 
tions  capacity  on  both  the  uplink  and  downlink  to  the  eNode  B  and  increased  cell 
tower  sector  capacity  and  promises  to  reduce  messaging  latency.  LTE  supports  Inter¬ 
net  protocol-based  traffic  with  end-to-end  quality  of  service  (QoS)  features.  The  first 
deployed  version  of  LTE,  Release  8,  which  became  available  in  2011,  provides  data  ser¬ 
vices  only.  Later  versions  of  LTE  will  provide  integrated  voice  and  data  services.  When 
LTE  voice  becomes  available,  voice  traffic  can  be  provided  using  several  implementa¬ 
tion  options.  One  of  these  will  be  voice  over  Internet  protocol  (VoIP). 

It  is  interesting  to  note  that  commercial  telecommunications  carriers  have  not  yet 
deployed  an  integrated  voice  and  data  LTE  network.  It  has  taken  longer  than  originally 
anticipated  for  LTE  developers  to  perfect  such  a  solution,  but  commercial  industry  is 
still  working  on  one.  Some  industry  observers  have  stated  that  such  a  solution  will 
be  available  in  Release  10  of  LTE.  At  one  point,  VoIP  was  envisioned  for  voice  com- 
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munications  using  WNW  on  the  JTRS  GMR,  but  this  possibility  now  seems  remote 
because  of  the  limited  communications  capacity  available  in  WNW  networks  with 
mobile  nodes. 

Figure  1.6  illustrates  the  rack-mounted  equipment  for  a  radio  base  station  (RBS) 
6201  LTE  eNode  B,  the  base  station  produced  by  Motorola.  Not  shown  are  the  anten¬ 
nas  that  would  also  be  a  part  of  the  system.  Smaller,  more  compact  eNode  B  terminals 
that  are  less  than  one-quarter  the  size  of  the  terminal  shown  in  the  figure  are  now 
available.  The  eNode  B  is  the  central  hub  of  the  wireless  portion  of  the  LTE  network. 
It  manages  all  wireless  network  connections  to  user  equipment  and  has  the  capacity 
to  control  up  to  200  users  per  cell  in  5  MFiz  of  spectrum.  So  if  a  single  eNode  B  has 
access  to  20  MFiz  of  spectrum,  it  can  manage  up  to  800  users  per  cell.  In  addition, 
if  the  eNode  B  employs  sector  antennas  and  provides  three  independent  sectors,  each 
using  an  independent  20  MFiz  block  of  spectrum,  it  would  be  able  to  support  up  to 
2,400  users. 

It  is  interesting  to  compare  the  sizes  of  the  wireless  networks  that  can  be  sup¬ 
ported  by  LTE  and  JTRS  WNW.  Although  WNW  was  originally  designed  to  support 
networks  with  as  many  as  200  nodes,  it  has  been  demonstrated  to  work  effectively  only 

Figure  1.6 
LTE  Base  Station 


SOURCE:  Motorola,  "Long  Term  Evolution 
(LTE):  A  Technical  Overview,"  Technical 
White  Paper,  2007. 
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with  networks  with  a  maximum  of  30  nodes  (as  of  2012).  In  contrast,  the  LTE  RAN 
can  easily  support  network  sizes  of  over  2,000  nodes.  LTE  does  this  by  employing 
directional  or  sector  antennas  and  using  wide  20  MHz  spectrum  blocks.  Nevertheless, 
one  can  see  that  an  LTE  network  at  the  radio  access  layer  can  scale  up  to  many  more 
users  or  nodes  than  WNW. 

When  comparing  LTE  RAN  equipment  to  JTRS  GMR  hardware,  it  is  appropri¬ 
ate  to  consider  both  the  end-user  device  (mobile  phone)  and  the  eNode  B  terminal. 
On  one  hand,  LTE  end-user  devices  are  much  smaller  than  GMR  terminals.  On  the 
other  hand,  a  typical  eNode  B  base  station,  as  indicated  in  Figure  1.6,  may  be  sev¬ 
eral  times  larger  than  a  complete  suite  of  GMR  equipment.  Partly  because  the  JTRS 
WNW  network  is  a  peer-to-peer  network,  all  nodes  are  of  equal  size,  whereas  in  an 
LTE  network,  the  functional  capability  of  end-user  devices  and  eNode  B  terminals  is 
significantly  different  and  so  are  their  sizes,  their  power  levels,  etc.  Network  control 
functions  are  all  housed  in  the  eNode  B  terminal.  In  contrast,  the  end-user  equipment 
or  mobile  phone  has  much  less  intelligence  built  into  it  and  is  designed  to  provide  the 
end  user  with  applications  and  middleware  to  support  user  interface  and  information 
display  functions. 

LTE  Evolved  Packet  System 

The  LTE  communications  architecture  is  based  on  a  set  of  standards  that  was  devel¬ 
oped  by  an  international  organization  called  3GPP.  Below,  we  describe  how  this  orga¬ 
nization  functions  and  how  it  led  to  the  development  of  LTE.  In  developing  LTE,  the 
3GPP  specified  a  new  packet  core,  the  evolved  packet  core  (EPC)  network  architecture. 
The  EPC  architecture  is  simpler  than  the  third-generation  (3G)  core  network  design 
and  has  fewer  network  elements,  with  simpler  functionality,  improved  redundancy, 
and  improved  interoperability,  which  allows  for  connections  and  handover  to  networks 
using  other  fixed  line  and  wireless  access  technologies. 

An  overview  of  the  LTE  system  architecture  evolution  (SAE)  network  architec¬ 
ture  is  shown  in  Figure  1.7.  For  the  sake  of  brevity,  we  do  not  delve  into  all  the  details 
of  LTE  networks.  Instead,  we  provide  an  overview  of  LTE  networks  and  describe  the 
key  network  nodes  and  functions  they  perform.  Note  that  user  equipment  is  not  shown 
in  the  figure.  A  voice  or  data  call  is  started  by  user  equipment  that  establishes  a  wire¬ 
less  connection  to  an  eNode  B,  which  is  indicated  with  the  antenna  structure  in  the 
figure.  Together,  the  LTE  wireless  and  backhaul  networks  are  known  as  the  evolved 
packet  system  (EPS). 

The  primary  nodes  of  the  backhaul  network  are  the  SGW,  the  packet  data  net¬ 
work  (PDN)  gateway,  and  the  mobility  management  entity  (MME).  The  SGW  serves 
as  the  local  anchor  in  the  backhaul  network  for  Internet  protocol  traffic  for  individual 
UE.  All  Internet  protocol  traffic  from  external  networks  is  first  sent  to  the  SGW.  The 
SGW,  in  coordination  with  the  MME,  maintains  awareness  of  exactly  which  eNode 
B  the  UE  is  connected  to  and  then  routes  Internet  protocol  packets  to  that  node.  The 
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Figure  1.7 

LTE/SAE  Backhaul  Network  Design 


SOURCE:  wray  castle  Ltd.,  "LTE  Voice  Options  and  Operations,"  Bridge  Mill,  Stramongate,  Kendal,  UK, 
2012. 
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MME  is  the  central  control  node  for  wireless  UE  in  the  LTE  network  and  manages  its 
connection  status  to  individual  eNode  Bs.  For  example,  when  UE  moves  away  from 
one  eNode  B  and  closer  to  another  eNode  B,  the  MME  will  reassign  the  UE  to  the 
second,  now  closer,  eNode  B.  In  other  words,  the  MME  manages  the  control  plane  of 
the  wireless  LTE  network.  The  protocols  running  between  the  UE  and  the  control  net¬ 
work  are  known  as  the  nonaccess  stratum  (NAS)  protocols.  The  MME  also  manages 
specific  Internet  protocol  connections  to  user  equipment  and  can  set  up  and  establish 
individual  Internet  protocol  connections  as  well  as  tear  them  down  (these  functions  are 
known  as  EPS  bearer  control). 

The  PDN  gateway  is  responsible  for  allocating  Internet  protocol  addresses  to 
UE,  QoS  enforcement,  and  flow-based  charging.  It  is  responsible  for  the  filtering  user 
downlink  Internet  protocol  packets  into  the  different  QoS-based  EPS  bearers. 

Two  other  important  nodes  in  the  LTE  backhaul  network  are  the  policy  control 
and  charging  rules  function  (PCRF)  and  the  home  subscriber  server  (HSS),  which 
are  also  shown  in  Figure  1.7.  The  EISS  contains  user  LTE  network  subscription  data 
such  as  the  user’s  EPS-subscribed  QoS  profile  and  roaming  access  restrictions.  It  also 
holds  information  about  the  PDNs  to  which  the  user  can  connect.  In  addition,  the 
HSS  holds  dynamic  information  such  as  the  identity  of  the  MME  to  which  the  user 
is  currently  attached  or  registered.  Finally,  the  HSS  may  also  be  integrated  with  the 
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LTE  network  authentication  center  (AUC),  which  performs  user  authentication  and 
network  access  control  functions.6 

The  PRCF  is  responsible  for  policy  control  decisionmaking.  This  server  maintains 
subscriber  account  information  also  held  by  the  HSS  and  ensures  that  individual  sub¬ 
scribers  gain  access  to  network  resources  and  are  assigned  the  EPS  barriers  with  QoS 
levels  that  are  consistent  with  their  subscriber  account  privileges. 

LTE  Security  Architecture 

The  LTE  security  architecture  is  shown  in  Figure  1.8.  The  figure  indicates  the  key 
nodes  performing  security  functions  as  well  as  the  links  or  interfaces  between  these 
nodes  including  links  to  the  UE  (indicated  as  a  cell  phone  in  the  figure). 

The  MME  and  HSS  provide  the  core  security  functions  for  the  LTE  network  for 
both  signaling  and  user  data.  When  a  UE  attaches  to  the  network,  it  is  authenticated 
by  the  network  by  verifying  the  user’s  subscriber  data  held  at  the  HSS.  Subscriber  data 
are  then  forwarded  to  the  MME  that  manages  the  initial  UE  network  attachment. 
When  the  UE  attaches  to  the  network,  the  MME  also  provides  security  keys  for  pro¬ 
tecting  all  EPS  bearers  established  for  UE  communications.  All  data  that  go  over  the 
RAN  are  encrypted  in  the  LTE  network. 


Figure  1.8 

LTE/SAE  Security  Architecture 
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Figure  1.8  shows  the  encryption  algorithms  that  can  be  used  to  protect  the  net¬ 
work  control  plane  and  user  data.  These  encryption  capabilities  operate  between  the 
UE  and  the  eNode  B  as  well  as  between  the  other  key  LTE  network  nodes  shown  in 
the  figure. 

LTE  Software  Architecture 

In  contrast  to  the  JTRS  software  architecture,  there  is  no  single  LTE/SAE  software 
architecture.  Furthermore,  LTE/SAE  is  not  software-defined  and  competing  vendors 
do  not  share  software  code.  Instead,  vendor  products  are  built  on  competing  chip  sets. 
Equipment  vendors  develop  products  on  specific  chip  sets  using  their  own  proprietary 
software  code,  which  may,  in  some  cases,  use  open  source  software  at  specific  layers  or 
in  specific  parts  of  their  products  (for  example  some  vendor  products  may  be  based  on 
the  open  source  Linux  operating  system).  Vendors  can  distinguish  their  products  on 
the  basis  of  user  interface,  look  and  feel,  applications,  etc.  Because  there  is  no  single 
code  base  for  LTE  network  equipment,  LTE  network  interoperability  is  ensured  by 
testing  vendor  products. 

During  the  course  of  this  study,  RAND  interviewed  a  number  of  industry  par¬ 
ticipants  from  major  cell  phone  network  infrastructure  providers.  These  interviews 
revealed  that  cell  phone  infrastructure  providers  perform  a  key  function  in  the  indus¬ 
try  by  testing  user  equipment  handsets  and  other  UE  equipment  for  compatibility  with 
their  own  network  infrastructure  products  (such  as  eNode  B  terminals).  In  addition, 
they  test  their  own  network  infrastructure  products  for  interoperability  and  compat¬ 
ibility  with  the  products  developed  by  competing  infrastructure  providers.  RAND 
found  that  industry  participants  conduct  extensive  testing  to  ensure  that  their  equip¬ 
ment  will  be  interoperable  and  conforms  to  3GPP  standards. 

LTE  Network  and  User  Equipment  Deployments 
Commercial  Network 

The  first  commercial  LTE  networks  were  reportedly  deployed  in  2009.7  Since  that 
time,  a  number  of  LTE  network  deployments  occurred  around  the  world.  U.S.  cel¬ 
lular  telephone  network  service  providers  are  in  the  forefront  of  the  transition  to  LTE. 
Verizon  wireless  has  deployed  LTE  in  over  470  market  areas  throughout  the  United 
States  as  of  the  fourth  quarter  of  2012. 8  AT&T  wireless  has  deployed  LTE  in  over  135 
markets  in  the  United  States  as  of  December  2012.9  Sprint  had  deployed  LTE  into 
49  markets  by  the  end  of  2012  and  promises  additional  deployments  into  150  addi- 


7  See  “LTE  (telecommunication),”  Wikipedia,  undated. 

8  See  Verizon,  “LTE  Information  Center,”  undated. 

9  “AT&T  Expands  LTE  Network  to  New  Markets,”  LTE  World,  December  21,  2012 
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tional  markets  soon.10  Many  wireless  carriers  in  Europe  and  Asia  have  also  aggressively 
started  to  deploy  LTE  in  their  coverage  areas. 

DoD  Deployments 

The  DoD  is  also  exploring  the  use  of  LTE  networks.  Commercial  off-the-shelf 
(COTS)-based  communications  technologies,  including  4G/LTE  broadband  systems, 
may  provide  government  users  with  high-capacity  mobile  communications  that  com¬ 
mercial  users  have  come  to  expect  from  wireless  broadband  networks.  In  addition, 
commercial  industry’s  faster  innovation  time  lines  are  often  shorter  than  the  longer 
acquisition  schedules  usually  required  to  develop  proprietary  special-purpose  govern¬ 
ment  communications  systems.  Furthermore,  if  COTS  systems  and  technologies  can 
be  leveraged  by  the  DoD,  they  may  provide  government  users  with  the  latest  advanced 
capabilities  as  they  are  introduced  in  the  commercial  markets.* 11 

The  capability  differences  between  tactical  military  communications  and  com¬ 
mercial  COTS  communications  have  appeared  to  diverge  over  the  last  two  decades, 
leaving  many  in  the  military  to  wonder  why  they  cannot  achieve  the  same  capabilities 
with  the  equipment  they  use  on  the  battlefield  as  what  they  have  become  accustomed 
to  in  civilian  life. 

Figure  1.9  shows  that  developers  of  commercial  waveforms  have  incrementally 
improved  commercial  mobile  networks  over  time  and  that  commercial  firms  have 
invested  significant  R&D  resources  into  this  progression.  Below,  we  examine  the 
resources  that  have  gone  into  this  development. 

Figure  1 .9  also  shows  how  waveforms  developed  for  the  commercial  mobile  wire¬ 
less  cellular  industry  have  evolved  at  least  as  “fast”  as  military  waveforms  (in  terms  of 
numbers  of  waveforms  developed),  and  perhaps  “faster”  in  terms  of  peak  data  rates. 
However,  it  is  important  to  note  that  the  difference  in  performance  growth  shown  in 
the  figure  does  not  really  compare  equivalent  capabilities.  The  commercial  waveforms 
shown  in  the  figure  do  not  have  the  security  features  that  military  waveforms  do.  In 
addition,  as  we  alluded  to  above,  the  JTRS  GMR  and  WNW  architecture  is  infra¬ 
structureless,  which  means  that  it  is  more  survivable  and  adaptive  than  the  LTE  archi¬ 
tecture  that  relies  on  fixed  cell  phone  tower  nodes.  If  a  cell  phone  tower  is  destroyed, 
cell  phone  communications  will  not  be  available  in  the  coverage  area  of  that  tower, 
unless  overlapping  coverage  is  available  from  another  tower  that  is  not  damaged. 

Nevertheless,  it  may  be  possible  to  adapt  or  modify  the  LTE  architecture  so  that 
it  can  meet  military  requirements.  Furthermore,  using  4G/LTE  in  a  military  applica¬ 
tion  would  not  be  the  first  major  use  of  COTS  technology  on  the  battlefield.  The  DoD 


10  Michelle  Maisto,  “Sprint  LTE  Network  Reaches  49  Markets,  150  More  Coming  Soon,”  eWeek,  December  20, 
2012 

11  Douglas  C.  Smith,  “Creating  Opportunities  through  Improved  Government  Spectrum  Efficiency,”  Testi- 
mony  before  the  House  Energy  and  Commerce  Committee,  Subcommittee  on  Communications  and  Technol- 
ogy,  Washington,  D.C.,  September  13,  2012a. 
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Figure  1.9 

The  Data  Capacity  Gap:  Tactical  Versus  Commercial  Communications 


SOURCE:  A.  R.  Hartman  et  al.,  "4G  LTE  Wireless  Solutions  for  DoD  Systems,"  Military  Communications 
Conference,  201 1 . 
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has  transitioned  to  COTS  technologies  when  government-funded  proprietary  systems 
failed  to  deliver  capabilities  similar  to  their  civilian  counterparts.  Commercial  laptop 
computers  and  tablet  devices  such  as  Apple’s  iPad  are  commonly  used  throughout  the 
U.S.  military.  “In  years  past,  the  Pentagon  probably  would  have  spent  billions  of  dol¬ 
lars  creating  its  own  custom  devices,  but  modern  technology  offers  a  much  cheaper 
alternative.”12 

Using  this  4G  technology  has  many  potential  advantages.  First,  the  technology 
development  life  cycles  of  commercial  cellular  systems  are  much  faster  than  those  of 
current  military  systems.  By  leveraging  the  continual  and  extensive  R&D  investment 
from  the  commercial  sector,  the  military  can  remain  on  the  leading  edge  of  advances 
in  waveform  standards,  network  equipment,  and  handset  equipment.  For  no  extra  cost, 
the  military  gains  the  benefit  of  the  commercial  rapid  technology  life  cycles  and  low 
cost  factors.  Second,  “use  of  commercial  wireless  broadband  technologies,  when  appro¬ 
priate  to  meet  mission  needs,  can  allow  military  and  other  Federal  government  opera¬ 
tions  to  use  spectrum  even  more  efficiently  and  cost-effectively.”13  Finally,  by  using  the 
same  waveform,  standards,  and  similar  equipment,  military  forces  can  ensure  interop¬ 
erability  with  civilian  forces  that  they  may  work  alongside  during  homeland  defense 
and  natural  disaster  situations. 


12  W.  J.  Hennigan,  “Taking  iPads  into  Battle,”  Los  Angeles  Times ,  September  25,  2011. 

13  Smith,  2012a. 
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Although  all  services  are  currently  testing  potential  4G/LTE  applications,  the 
U.S.  Navy  was  deploying  a  test  system  in  early  2013.  A  4G  LTE  network  was  to 
be  operationally  tested  on  the  USS  Kearsarge,  the  USS  San  Antonio ,  and  the  USS 
Whidbey  Island  beginning  in  2013.  A  team  led  by  Oceus  Networks,  an  offshoot  of 
Ericsson,  developed  this  system,  also  known  as  the  NAVY  4G  /LTE  afloat.  The  base¬ 
line  capabilities  of  this  network  will  provide  high-speed  Internet  at  8-15  Mbps.  “The 
4G  systems  on  all  three  ships  will  have  a  line-of-sight  range  of  15-20  miles,  which  can 
be  extended  to  30  miles  with  the  use  of  a  base  station  on  a  Marine  H-l  helicopter.”14 

For  the  first  deployment  with  the  USS  Kearsarge,  200  Marines  will  be  outfitted 
with  Android  handsets  with  standard  smart  phone  capabilities  such  as  voice,  text,  data, 
full  motion  video  (FMV),  navigation,  blue-force  tracking,  biometrics,  and  a  high-res¬ 
olution  camera.  The  built-in  Global  Positioning  System  (GPS)  capabilities  paired  with 
the  blue-force  tracking  application  within  the  phones  will  allow  the  deployed  Marines 
and  their  commanders  to  track  each  other’s  positions  in  real  time.  The  first  versions 
will  be  authorized  to  transmit  only  unclassified  information,  but  Naval  Air  Systems 
Command  is  “working  with  the  NSA  to  create  a  solution  that  will  allow  the  phones  to 
transmit  classified  data.”15  The  system  will  interface  directly  with  ships’  existing  Auto¬ 
mated  Digital  Network  System  (ADNS)  and  Distributed  Common  Ground  Station- 
Intelligence  Community  (DCGS-IC)  enterprise  services. 

The  deployed  test  equipment  has  the  capacity  to  support  up  to  3,500  Marines  and 
sailors  deployed  with  an  Amphibious  Ready  Group  (ARG).  Using  the  4G/LTE  wave¬ 
form,  similar  interoperable  baseline  systems  in  an  airborne  configuration  can  support 
communications  up  to  60+  miles.  This  capability,  combined  with  session  handover 
and  scalable  capabilities,  could  aggregate  into  a  virtually  unlimited  coverage  area.  Such 
a  network  would  allow  a  military  unit  conducting  combat  operations  in  an  austere 
environment  to  have  connectivity  to  each  other  similar  to  what  they  experience  with 
their  personal  smartphones  in  civilian  life.  Multiple  base  stations  on  airborne  platforms 
could  theoretically  provide  countrywide  communications.  The  ability  to  create  the 
functionality  of  a  full  cellular  network  in  a  single  unit  on  a  mobile  platform  overcomes 
limitations  of  applications  with  fixed  switching  equipment.  The  mobile  network  equip¬ 
ment  application  “can  be  placed  aboard  ships,  installed  in  tactical  warfighter  vehicles, 
mounted  on  UASs  [unmanned  aircraft  systems]  and  other  aerial  vehicles,  and/or  be 
soldier  back-packed.”16 

The  Navy’s  4G/LTE  Afloat  project  designates  4G  as  a  “mission  critical  require¬ 
ment”  for  the  Counter-Piracy  Task  Force,  which  mostly  operates  off  the  Horn  of  Afri- 


14  Bob  Brewin,  “Naval  Air  Systems  Command  Plans  4G  Cell  Service  aboard  Ships,”  Nextgov.com ,  April  12, 
2012a. 

15  Brewin,  2012a. 

16  Smith,  2012a. 
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ca.17  This  network  could  be  used  to  send  critical  intelligence  including  airborne  FMV 
to  Marines  before  they  boarded  vessels  captured  by  pirates  or  terrorists.  It  could  also 
be  used  to  relay  critical  real-time  intelligence  back  to  the  ARG  command  element  on 
the  USS  Kearsarge. 

Although  the  Navy’s  4G/LTE  Afloat  project  is  the  first  to  be  tested  operationally, 
many  other  similar  efforts  are  currently  ongoing.  Other  test  applications  of  4G  LTE 
within  the  DoD  include  the  following: 

•  A  Navy  pilot  program  was  scheduled  to  go  on  a  UH-1  in  November  2012.  In  this 
application,  a  base  station,  eNodeB,  and  an  EPC  was  to  be  carried  on  a  UH-1. 
The  intent  of  this  demonstration  was  to  create  a  full-blown  4G/LTE  airborne  net¬ 
work  for  anyone  on  land  or  sea  with  line  of  sight  to  connect. 

•  Oceus  Networks  is  working  with  the  Gorgon  Stare  program  to  fly  on  that  plat¬ 
form.  This  test  will  investigate  the  possibility  of  replacing  or  supplementing  ele¬ 
ments  of  the  current  tactical  datalink. 

•  Teams  are  working  with  the  Joint  Special  Operations  Command  (JSOC),  test¬ 
ing  on  a  Pilatus  PC-12  and  a  C-130  wing  pod  to  provide  LTE  coverage  from  the 
air.  If  the  initial  test  is  successful,  it  may  become  a  Joint  Capability  Technology 
Demonstration  (JCTD)  initiative. 

•  Teams  are  working  with  elements  of  the  U.S.  Air  Force  to  determine  whether 
multiple  MQ-9  Reaper  UAS  systems  can  be  flown  around  a  large  geographic  area 
to  provide  countrywide  coverage. 

•  Teams  are  working  with  the  Air  Force  BIG  SAFARI  to  test  the  LTE  application 
on  smaller  air  platforms,  with  receivers  on  UASs.  “BIG  SAFARI  is  the  USAF’s 
program  office  responsible  for  sustainment  and  modification  of  specialized  special 
mission  aircraft.”18 

•  Research  is  also  being  conducted,  especially  in  air  applications,  to  model  and 
study  performance,  fading  effects,  and  Doppler  effects  for  airborne  applications. 

The  DoD  is  not  the  only  government  agency  looking  at  potential  uses  for  this  type 
of  mobile  network.  The  FCC  is  investigating  the  possibilities  of  putting  a  base  station 
and  core  network  on  a  weather  balloon  at  100,000  feet  to  provide  first-responder  state¬ 
wide  coverage  for  civilian  and  military  applications.  The  FCC  has  begun  a  major  effort 
to  study  how  deployable  aerial  communications  architecture  (DACA)  “can  restore  the 
communications  capabilities  of  first  responders  shortly  after  the  occurrence  of  a  major 
natural  disaster  or  terrorist  attack.”19  Equipment  similar  to  that  currently  being  tested 
under  the  Navy  program  on  the  UH-1  could  fly  on  weather  balloons  or  high- altitude/ 


17  Neal  Ungerleider,  “The  4G  System  That  Powers  The  Navy’s  Pirate  Fights,”  Fast  Company ,  April  19,  2012. 

18  Federation  of  American  Scientists,  “Big  Safari,”  FAS  Intelligence  Research  Program,  July  30,  1997. 

19  Smith,  2012a. 
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long-endurance  UASs.  This  airborne  network  could  provide  emergency  communica¬ 
tions  to  local,  state,  and  federal  government  agencies  and  personnel  after  a  major  disas¬ 
ter  or  terrorist  attack  if  the  terrestrial  communications  infrastructures  were  disrupted, 
damaged,  or  destroyed.  4G  LTE  equipment  vendors  are  developing  rapidly  deployable 
aerial  communications  systems  that  can  provide  immediate  broadband  communica¬ 
tions  to  disaster  areas.20 

4G/LTE  has  the  potential  to  provide  solutions  to  many  gaps  in  tactical  commu¬ 
nication  architectures,  but  there  are  limitations.  Many  fielded  tactical  networks  in  the 
military  differ  from  their  civilian  counterparts  in  that  they  provide  secure  communi¬ 
cations  and  in  some  cases  antijam  capabilities.  Efforts  are  currently  ongoing  with  4G/ 
LTE  equipment  and  standards  to  meet  secure  communications  requirements.  Creat¬ 
ing  an  antijam  capability  outright  through  techniques  such  as  frequency  hopping  may 
prove  challenging — without  losing  commonality  and  interoperability  with  civilian 
4G/LTE  waveforms  and  equipment. 

Although  commercial  4G/LTE  systems  have  many  builtin  security  features,  the 
Defense  Information  Systems  Agency  (DISA)  and  NSA  have  determined  that  more 
work  is  required  to  meet  military  security  requirements.  The  Naval  Air  Systems  Com¬ 
mand  is  working  on  standards  to  scramble  or  encrypt  any  storage  that  exists  on  the 
phone  to  facilitate  NSA  requirements.  DISA  and  NSA  understand  that  it  is  too  dif¬ 
ficult  to  keep  up  with  all  the  commercial  LTE  software  and  hardware  releases.  NSA  is 
in  the  process  of  giving  direction  on  what  it  would  like  to  see  as  far  as  password  aging 
and  password  complexity,  resulting  in  standards  on  encryption  of  data  in  storage  on 
LTE  devices.  The  Navy  pilot  program  has  been  through  Joint  Interoperability  Test 
Command  (JITC)  certification.  The  current  4G/LTE  solution  has  completed  IA  cer¬ 
tification  on  land  and  mobile  devices  and  is  now  progressing  toward  maritime  and  air 
certification. 

4G/LTE  is  a  commercial  waveform  and  has  limited  frequency-hopping  capabili¬ 
ties.  The  Defense  Advanced  Research  Projects  Agency  (DARPA)  has  investigated  simi¬ 
lar  antijam  waveforms,  but  the  derivative  product  is  a  highly  proprietary  waveform  that 
requires  proprietary  handsets  and  base  stations.  Although  this  approach  has  unique 
applications,  it  is  a  separate  effort  and  would  not  benefit  from  the  massive  amounts  of 
civilian  research  and  development  spent  on  4G  LTE.  There  are  efforts  to  apply  tech¬ 
niques  previously  developed  through  years  of  testing  to  counter  GPS  jammers.  These 
techniques  have  proven  successful  in  allowing  the  military  unfettered  use  of  GPS  in 
jamming  environments.  Although  there  are  some  very  basic  differences  between  GPS 
and  4G/LTE  frequencies  and  waveforms,  the  techniques  and  procedures  developed  to 
counter  GPS  jamming  show  promise  for  the  use  of  4G/LTE  waveforms  and  equipment 
in  jamming  environments.  These  techniques  are  planned  for  future  integration  into 
this  test  application. 
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Comparison  of  Development  Approaches  and  Organizational 
Structures 

In  this  section,  we  compare  the  development  approaches  and  organizational  structures 
responsible  for  the  development  of  the  JTRS  family  of  programs  and  for  LTE. 

JTRS  Development  Organizations 

The  JTRS  program  consists  of  a  family  of  software-defined  radios  and  waveforms 
designed  to  be  easily  interoperable.  All  JTRS  radio  hardware  is  SCA-compatible, 
meaning  that  all  hardware  shares  a  common  software  framework  for  waveform  devel¬ 
opment  and  integration.  The  use  of  SCA  on  base  radio  hardware  will  enable  a  common 
set  of  waveforms  to  be  ported  to  different  radio  hardware. 

Here,  we  describe  the  organizational  structure  for  the  family  of  JTRS  programs 
that  was  responsible  for  the  development  of  all  JTRS  SDRs  from  2005  to  2012.  The 
JTRS  organizational  structure  is  shown  in  Figure  1.10.  All  JTRS  programs  of  record 
were  placed  under  the  oversight  of  a  JTRS  Joint  Program  Executive  Office  (JPEO). 
JTRS  radio  hardware  and  waveform  software  are  developed  independently  under  the 
oversight  of  different  program  offices  and,  in  some  cases,  different  contractor  teams 
(for  example,  GMR  and  WNW  are  under  the  same  prime  contractor  but  different  sub¬ 
contractors).  WNW  was  developed  by  the  JTRS  NED  program  office,  and,  as  noted 
above,  WNW  is  based  on  the  JTRS  SCA.  The  development  of  other  JTRS  network¬ 
ing  waveforms  was  also  the  responsibility  of  the  JTRS  NED  program.  The  SCA  was 
developed  by  an  industry  consortium  with  strong  leadership  from  the  JTRS  JPEO.  The 
JTRS  GMR  program  office  was  responsible  for  developing  radio  hardware  designed  to 
run  WNW  and  was  also  responsible  for  GMR  field  testing. 

The  JTRS  JPEO  was  responsible  for  ensuring  coordination  between  JTRS  pro¬ 
grams.  Although  WNW  was  developed  by  NED,  and  GMR  was  developed  by  the 


Figure  1.10 

JTRS  Organizational  Structure 
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GMR  program,  the  JPEO  ensured  that  the  products  developed  within  the  individual 
programs  were  interoperable  and  compatible  with  each  other. 

A  key  architectural  construct  that  the  JPEO  itself  was  responsible  for  was  the 
SCA  that  was  to  be  used  by  all  JTRS  programs.  The  SCA  provided  a  framework  for 
software  development  that  was  used  by  both  the  hardware  and  software  programs.  In 
this  regard,  it  is  important  to  note  that  JTRS  hardware  programs  (i.e.,  the  programs 
that  actually  built  the  radios)  also  were  responsible  for  developing  the  operating  sys¬ 
tems  for  these  radios.  JTRS  radios  are  software-defined,  which  means  that  they  resem¬ 
ble  personal  computers  (PCs)  in  many  respects.  Like  a  PC,  a  JTRS  radio  would  have 
an  operating  system  on  which  waveforms  could  be  loaded.  The  user  can  load  applica¬ 
tions  on  to  the  PC,  just  as  would  be  done  with  a  Windows  PC  using  Windows  OS. 
The  SCA  is  the  framework  the  radio  OSs  had  to  conform  to  as  well  as  the  waveforms. 
It  helped  to  ensure  that  waveforms  would  be  portable  across  hardware  platforms  (could 
be  designed  and  developed  on  one  hardware  platform  of  the  JTRS  family  of  radios  and 
ported  to  other  platforms  in  the  family). 

The  development  of  JTRS  waveforms  was  the  responsibility  of  the  JTRS  NED 
program,  and  the  development  of  JTRS  radios  was  the  responsibility  of  a  number  of 
programs  as  indicated  in  the  figure.  The  particular  waveform  we  focus  on  in  this  analy¬ 
sis  is  WNW,  which  is  the  most  complex  waveform  developed  by  any  of  the  JTRS  pro¬ 
grams.  Originally,  WNW  was  developed  to  run  on  only  a  single  JTRS  radio  (GMR), 
which  was  the  most  complex  and  costly  hardware  platform  developed  in  the  JTRS 
family  of  radios. 

One  potential  challenge  associated  with  the  JTRS  organizational  structure  is 
ensuring  that  products  developed  by  different  programs  are  compatible  and  work 
together  as  originally  envisioned.  At  the  time  the  JTRS  program  was  created,  the  PC 
was  the  dominant  consumer  computing  platform  in  which  hardware  and  software 
development  was  separated  in  commercial  industry.  (Microsoft  made  the  software,  and 
Intel  and  other  original  equipment  manufacturers  [OEMs]  made  the  hardware.)  The 
advantage  of  this  approach  was  that  competition  among  hardware  OEMs  reduced  the 
price  of  PCs  and  enabled  them  to  dominate  the  market  for  low-cost  computers.  How¬ 
ever,  as  the  Windows  architecture  aged,  and  as  other  companies  introduced  smaller, 
more  mobile  computing  platforms  that  were  more  power  efficient  than  PCs,  they 
began  to  be  preferred  by  consumers.  These  new  mobile  computing  devices  also  turned 
out  to  have  radio  capabilities  and  have  become  increasingly  important  parts  of  the  con¬ 
sumer  electronics  market.  These  include  the  Apple  iPhone,  mobile  devices  operating 
the  Android  OS,  as  well  as  BlackBerrys.  Therefore,  for  commercial  wireless  and  mobile 
devices,  it  appears  that  much  tighter  integration  is  required  for  hardware  and  software, 
which  is  in  contrast  to  the  JTRS  organizational  structure. 

The  JTRS  organizational  structure  could  work  if  hardware  and  software  develop¬ 
ment  could  be  easily  separated  and  if  it  could  be  developed  effectively  in  separate  orga¬ 
nizations.  However,  it  should  also  be  pointed  out  here  that  the  JTRS  SDR  architecture, 
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which  is  based  on  the  SCA,  is  in  some  important  ways  different  from  the  PC  hardware 
and  software  architecture.  Most  PCs  do  not  provide  wireless  communications  capabili¬ 
ties  and  do  not  require  specialized  antennas,  power  amplifiers,  and  A/D  converters,  as 
SDRs  do.  These  components,  which  are  unique  to  digital  radios,  make  it  more  difficult 
to  separate  the  “software  stack”  and  OS  functions  from  hardware.  Indeed,  this  is  per¬ 
haps  one  reason  why  commercial  industry  has  chosen  not  to  adopt  an  SDR  approach 
for  the  LTE  architecture. 

The  JTRS  program  has  encountered  significant  instability  and  several  program 
restructurings  as  a  result  of  cost  growth,  requirements  changes,  schedule  slips,  and 
difficulties  in  meeting  stated  or  implied  requirements.  These  have  led  to  a  subsequent 
Nunn-McCurdy  breach  for  the  JTRS  GMR  program.21  In  fact,  the  JTRS  program  has 
undergone  a  number  of  reorganizations  within  the  past  several  years  that  have  resulted 
in  the  following: 

•  cancelation  of  JTRS  GMR  program22 

•  cancelation  of  the  JTRS  AMF  program23 

•  phase-out  of  the  JTRS  JPEO  and  transition  to  the  Joint  Tactical  Networking 
Center  (JTNC)24 

•  phase-out  of  the  JTRS  NED  program  and  transition  to  the  Joint  Tactical  Net¬ 
work  (JTN)  program.  25 

In  addition  to  the  cancelations  and  phaseouts  mentioned  above,  restructuring 
efforts  also  resulted  in  the 

•  continuation  of  the  JTRS  HMS  program26 

•  continuation  of  the  JTRS  Multifunctional  Information  Distribution  System  pro¬ 
gram.27 


21  J.  Michael  Gilmore,  DoD  Operational  Test  and  Evaluation  FY 2011  Annual  Report,  Washington,  D.C.:  Depart¬ 
ment  of  Defense,  2011. 

22  JTRS  GMR-SAR,  “Selected  Acquisition  Report,”  Department  of  Defense,  December  31,  2011. 

23  Airborne  &  Maritime/Fixed  Station  Joint  Tactical  Radio  System  (AMF  JTRS),  “Selected  Acquisition  Report,” 
Department  of  Defense,  December  2012. 

24  Bob  Brewin,  “Pentagon  Shutters  Joint  Tactical  Radio  System  Program  Office,”  Nextgov.com,  August  1,  2012b. 

23  Barry  Rosenberg,  “From  Radios  to  Waveforms:  How  JTRS  Is  Remaking  Itself  as  JTNC,”  Defense  Systems, 
September  6,  2012. 

26  Rosenberg,  2012. 

27  Rosenberg,  2012. 
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As  noted  in  the  list  above,  the  JTRS  JPEO  was  formally  disestablished  in  July 
2012. 28  Before  this  decision,  the  JTRS  GMR  and  AMF  programs  were  canceled  in 
2011.  However,  it  appears  that  the  JTRS  networking  waveforms  developed  by  NED 
and  tested  on  several  prototype  JTRS  hardware  platforms  such  as  GMR  will  still  be 
fielded  and  used  on  other  SDRs  developed  by  other  vendors.  For  example,  the  SRW 
waveform  has  been  ported  to  the  Harris  PRC-1 17G,  and  WNW  may  be  ported  to 
several  other  industry-developed  SDR  platforms.  In  the  same  memorandum  that  dis¬ 
established  the  JTRS  JPEO,  the  JTRS  NED  program  was  renamed  the  JTN  program. 
Responsibility  for  the  development  and  maintenance  of  the  SCA  was  assigned  to  the 
JTN  program. 

All  of  these  organizational  changes  suggest  that  the  JTRS  program  was  troubled 
and  that  the  attempted  solutions  included,  in  part,  changes  to  the  program  structure. 
We  explore  this  question  in  more  detail  below. 

LTE  Development  Organizations 

LTE  is  a  family  of  systems  that  forms  an  integrated  cellular  communications  archi¬ 
tecture.  This  architecture  is  designed  so  that  system  products  from  different  vendors 
can  be  integrated  into  the  same  network  (i.e.,  they  are  interoperable).  LTE  is  a  family 
of  waveforms  and  architecture  for  next-generation  cellular  telephone  networks  that 
will  eventually  unify  data  and  voice  communications  networks  into  a  single  Internet 
protocol  network.  To  date,  it  is  the  most  advanced  and  complex  terrestrial  waveform 
development  in  the  commercial  world. 

LTE  includes  a  set  of  standards  developed  by  3GPP  for  mobile  3G  and  4G  mobile 
communications.  The  currently  fielded  standard  is  LTE  Release  8  (December  2008). 
The  mobile  standard  is  Evolved  UMTS  [Universal  Mobile  Telecommunications  Stan¬ 
dard]  Terrestrial  Radio  Access  (E-UTRA)  and  the  base  station  standard  is  Evolved 
UMTS  Terrestrial  Radio  Access  Network  (E-UTRAN).  Given  that  LTE  is  not  soft¬ 
ware-defined,  standards  apply  to  both  LTE  software  and  hardware. 

LTE  development  is  based  on  a  much  different  organizational  model  than 
the  one  for  the  JTRS  family  of  programs  discussed  in  the  preceding  section.  LTE  is 
being  developed  by  a  large  and  complex  international  organization-of-organizations 
known  as  3GPP.29  3GPP  “unites  six  telecommunications  standard  development 
organizations  .  .  .  known  as  ‘Organizational  Partners’  and  provides  their  members 
with  a  stable  environment  to  produce  .  .  .  reports  and  specifications  that  define  3GPP 
technologies.”30 

The  telecommunications  standard  development  organizations  involved  in  3GPP 

are: 


28  JTRS  GMR-SAR,  2011. 

29  3GPP,  “The  Mobile  Broadband  Standard,”  website,  undated-b. 

30  3GPP  “About  3GPP,”  website,  undated-a. 
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•  the  Association  of  Radio  Industries  and  Businesses,  Japan 

•  the  Alliance  for  Telecommunications  Industry  Solutions,  USA 

•  China  Communications  Standards  Association 

•  the  European  Telecommunications  Standards  Institute 

•  Telecommunications  Technology  Association,  Korea 

•  Telecommunication  Technology  Committee,  Japan. 

In  addition,  there  are  13  market  representation  partners  and  378  individual 
members. 

The  organizational  structure  of  3GPP  is  not  focused  on  developing  hardware  and 
software  products  but  rather  on  developing  technical  standards  that  can  then  be  used 
by  competing  firms  to  develop  hardware  and  software  products. 

The  3 GPP  organizational  structure  is  shown  in  Figure  1.11.  Industry  and  tech¬ 
nology  experts  from  individual  member  firms  form  technical  specification  groups  that 
develop  technical  specifications  for  new  systems  as  indicated  in  the  figure.  For  a  large, 
complex  system  such  as  FTE,  the  technical  specification  group  may  establish  sub¬ 
working  groups  that  develop  technical  standards  for  specific  parts  of  the  architecture 
design.  These  technical  specifications  are  reviewed  by  the  3GPP  organizational  part¬ 
ners  and  may  be  revised  accordingly.  Once  such  technical  specifications  are  approved 

Figure  1.11 

LTE  Organizational  Structure 


SOURCE:  3GPP,  "Third  Generation  Partnership  Project  (3GPP)  Partnership  Project  Description,"  April  12, 
1998. 
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they  can  be  used  in  the  design  of  new  telecommunications  systems.  Technical  experts 
who  are  not  employed  by  individual  3GPP  member  companies  or  who  are  not  mem¬ 
bers  of  3GPP  organizational  partner  committees  may  not  participate  in  3GPP  techni¬ 
cal  specification  working  groups.  One  key  value  in  being  a  member  firm  is  having  the 
ability  to  influence  the  technical  design  and  specification  of  next-generation  telecom¬ 
munications  systems. 

One  challenges  with  developing  a  set  of  standards  for  4G  cellular  networks  is 
achieving  cooperation  among  a  very  diverse  set  of  industry  firms  that  sometimes  oper¬ 
ate  and  compete  in  the  same  markets  and  sometimes  in  different  markets  around  the 
world.  Another  challenge  is  ensuring  that  the  standards  developed  by  the  organization 
lead  to  interoperable  products  because  a  common  software  platform  is  not  assumed  or 
used.  The  consensus-oriented  process  used  by  3GPP  helped  maintain  the  interoperabil¬ 
ity  of  a  system  based  on  components  from  multiple  competing  developers. 

The  3GPP  organization  grew  out  of  the  European  standards  development  process 
that  was  used  to  develop  technical  standards  for  the  second-generation  (2G)  digital 
mobile  phone  system,  Global  System  for  Mobile  Communications  (GSM).  The  GSM 
standard  was  developed  by  the  European  Telecommunications  Standards  Institute 
(ETSI),  which  is  an  independent,  nonprofit,  standardization  organization  composed 
of  equipment-makers  and  network  operators  in  Europe.  The  GSM  standard  was  widely 
adopted  in  Europe,  but  other  competing  2G  mobile  phone  standards  and  systems  were 
developed  and  deployed  in  Asia  and  the  United  States. 

At  the  conclusion  of  the  GSM  development,  ETSI  members  turned  their  atten¬ 
tion  to  the  development  of  technical  standards  for  a  new  3G  mobile  phone  system 
standard.  ETSI  and  other  Standards  Development  Organizations  (SDOs)  initiated  dis¬ 
cussions  on  development  of  a  UMTS  for  3G  mobile  phone  systems  that  could  be  used 
in  all  regions  of  the  world.  Such  a  standard  would  enhance  mobile  phone  interoper¬ 
ability  and  make  it  easier  for  users  to  roam  on  other  mobile  phone  networks  outside 
their  home  countries. 

In  1998,  a  group  of  SDOs  interested  in  UMTS  formed  3GPP  by  signing  the  3rd 
Generation  Partnership  Project  Agreement.  The  original  task  of  3GPP  was  to  produce 
technical  specifications  and  reports  for  UMTS.31  The  UMTS  was  developed  rapidly 
and  was  released  in  1999.  Later,  3GPP  was  given  the  task  of  maintaining  the  GSM 
standard  and,  later,  of  developing  LTE  technical  specifications. 

Summary 

The  organizational  structure  of  the  JTRS  family  of  programs  has  changed  at  least  three 
times  during  its  lifetime.  The  number  of  perturbations  in  the  organizational  struc¬ 
ture  because  of  program  instability,  uncontrolled  cost  growth,  and  schedule  delays  has 
contributed  to  subsequent  program  cancellations  and  restructuring.  Whether  these 


31  “UMTS  3G  History,”  Radio-Electronics.com ,  January  2,  2013. 
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restructuring  efforts  have  effectively  curtailed  cost  growth  and  program  instability 
remains  to  be  seen.  In  contrast,  the  LTE  development  has  proceeded  under  the  same 
3GPP  organizational  structure  for  the  entire  development.  Furthermore,  3GPP  has 
added  new  working  subgroups  as  required  that  focus  on  defining  standards  for  specific 
components  of  the  larger  architecture. 

It  should  also  be  noted  that  LTE  designers  changed  LTE  system  requirements 
for  the  initial  releases  (Release  8  and  9)  when  it  was  determined  that  some  require¬ 
ments  would  be  difficult  to  meet  (e.g.,  integrated  voice  and  data  on  the  same  backhaul 
network).32  These  changes  will  be  reflected  in  Release  10.  3GPP  may  also  be  establish¬ 
ing  a  related  working  subgroup  to  address  these  technical  challenges. 

In  summary,  LTE’s  organizational  structure  is  intentionally  designed  to  be  flex¬ 
ible  to  enable  the  organization  to  address  technical  challenges  as  they  arise.  In  con¬ 
trast,  it  is  not  clear  that  the  program  reorganizations  that  the  JTRS  program  has  gone 
through  have  resulted  in  an  organizational  structure  with  the  same  ability  to  deal  with 
unforeseen  technical  hurdles. 


A  Comparison  of  Technical  Performance  and  Development  Time  Lines 

In  this  section,  we  compare  the  technical  performance  characteristics  and  the  develop¬ 
ment  schedules  for  the  LTE  and  JTRS  GMR  and  WNW  communications  systems.  In 
particular,  we  investigate  the  questions  of  how  similar  or  different  these  systems  and 
waveforms  are,  and  whether  one  or  the  other  system  required  more  time  to  develop. 

Technical  Performance  Comparison 

Some  characteristics  of  the  two  systems  are  quite  different,  but  other  aspects  are  quite 
similar.  We  review  the  major  characteristics  of  each  system  below.  First,  we  note  that 
the  security  architectures  of  the  two  systems  are  quite  different,  as  is  the  protocol  by 
which  users  are  authenticated  in  the  two  networks.  Furthermore,  user  message  traffic 
is  prioritized  and  controlled  in  a  different  way  in  each  network.  The  LTE  architec¬ 
ture  includes  subscriber  account-based  QoS  channel  (or  EPS  bearer)  network  resource 
access  controls,  whereas  WNW  instead  employs  a  message-based  QoS  scheme,  where 
individual  users  can  independently  choose  the  priority  of  the  messages  they  send  (five 
priorities  are  available  to  select  from).  The  presence  of  the  latter  type  of  QoS  scheme 
is  not  surprising  in  a  peer-to-peer  network  that  has  no  centralized  control  authority. 
In  contrast,  in  the  LTE  architecture,  the  network  service  provider  can  deny  network 
resources  to  subscribers  according  to  its  own  policies  and  the  account  status  of  the 
subscriber.  As  indicated  in  the  section  comparing  development  approaches  and  struc- 


32  The  “cell  towers”  of  wireless  mobile  communications  networks  are  connected  to  other  the  cell  towers  and  the 
core  network  of  the  mobile  communications  provider  by  means  of  a  backhaul  network.  The  backhaul  network 
may  refer  to  a  wire  line  or  fiber  optic  network,  or  high-capacity  wireless  links. 
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tures,  the  LTE  network  has  a  hierarchical  structure  where  network  resource  access 
control  decisions  are  centrally  managed  by  the  HSS  and  PCRF  nodes.  And,  finally,  it 
is  important  to  note  that  user  bandwidth  usage  is  monitored  in  the  LTE  network,  so 
that  the  network  service  provider  can  bill  users  according  to  their  network  usage  (e.g., 
the  number  of  voice  minutes  or  data  packets  consumed  per  month).  In  contrast,  such 
user  monitoring  and  billing  functions  are  absent  from  the  JTRS  WNW  network. 

Despite  these  differences,  some  portions  of  the  wireless  communications  wave¬ 
forms  used  in  both  systems  are  quite  similar.  The  wireless  radio  frequency  interface  of 
WNW  and  LTE  uses  similar  radio  signal  modulation  schemes,  but  there  are  important 
differences  as  explained  below.  Further,  in  some  respects  the  LTE  waveform  is  more 
advanced  than  that  used  by  WNW.  The  WNW  wireless  communications  waveform 
uses  symmetric  links;  that  is,  the  radio  frequency  characteristics  of  the  uplink  and 
downlink  from  a  particular  GMR  terminal  are  the  same.  The  LTE  wireless  channel  is 
asymmetric.  Each  LTE  wireless  channel  is  defined  as  a  link  between  the  eNode  B  and 
the  UE.  The  LTE  downlink  is  defined  as  the  wireless  signal  that  is  transmitted  from 
the  eNode  B  to  the  UE,  whereas  the  LTE  uplink  is  defined  as  the  wireless  signal  from 
the  UE  to  the  eNode  B. 

Both  LTE  and  WNW  use  similar  orthogonal  frequency-division  multiplexing 
(OFDM)  or  orthogonal  frequency-division  multiple  access  (OFDMA)  waveforms  for 
the  downlink.33  OFDMA  is  spectrally  efficient  and  enables  multiple  carriers  to  be 
closely  spaced.  However,  LTE  uses  a  different  waveform  on  the  uplink,  single-carrier 
frequency  division  multiple  access  (SC-FDMA).  As  the  name  implies,  SC-FDMA  is  a 
channel  allocation  managing  communications  from  multiple  users  that  share  the  same 
frequency  band.  In  that  sense,  it  shares  many  characteristics  with  OFDMA.  However, 
SC-FDMA  is  more  power-efficient  than  OFDMA,  making  it  better  suited  for  smaller, 
power-limited  UE  devices  (such  as  mobile  phone  handsets). 

WNW  and  LTE  have  different  downlink  and  antenna  capabilities.  WNW 
assumes  the  use  of  a  single  L-band  antenna.  LTE  can  use  multiple  antennas  on  both 
the  downlink  and  uplink.  The  minimal  implementation  of  the  LTE  standard  assumes 
that  (1)  the  eNode  B  transmits  the  downlink  on  two  antennas  and  receives  the  uplink 
on  a  single  antenna  and  (2),  the  UE  receives  on  two  antennas  and  transmits  the  uplink 
on  a  single  antenna.  In  addition,  the  LTE  standard  allows  for  additional,  more  pow¬ 
erful  multiple-input  and  multiple- output  (MIMO)  capabilities  for  both  the  eNode  B 
and  UE,  as  indicated  in  Table  1.1.  Eventually,  both  2-by-2  and  4-by-4  MIMO  LTE 
terminals  will  be  possible  where  both  the  terminal  transmitter  and  receiver  can  use  two 
or  four  antennas.  MIMO  provides  several  advantages  for  wireless  communications.  It 
can  reduce  sensitivity  to  channel  noise  and  multipath  and  provides  increased  process¬ 
ing  gain  for  weak  signals  (although  it  requires  increased  computational  power  to  pro- 


33  L3  Communications,  Wideband  Networking  Waveform  OFDM  PHY,  undated. 
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Table  1.1 

Technical  Performance  Metrics 


LTE/SAE 

WNW 

LTE  Rel  8  User  Equipment 
Cat. 

1 

2  3 

Max. 

Channel  bandwidth(s) 
(MHz) 

1.4,  3.0,  5.0,  10.0, 

15.0,  20.0 

1.2,  3.0,  5.0 

Peak  rate:  uplink  (Mbps) 

5 

25  50 

75 

5 

Peak  rate:  downlink 
(Mbps) 

10 

50  100 

300 

5 

Node  join  (sec) 

100  ms 

<  60  sec  to  add  1  node  to 
network 

<  120  sec  to  add  8  nodes  to 
network 

Data  message  latency 
(ms) 

100  ms  data 

LTE  standard  for  voice  is  to  be 
determined 

100  ms 

75  ms  for  tactical  voice — later 
dropped 

Network  formation  (see)3 

n/a 

<  15  min  in  150  node  network 

Network  node  scalability 

1,000s — at  wireless  tier 
10,000s 

No  formal  requirement,  but 
implied  need  to  support  -1,500 
nodes  (FCS-GAO) 

30  nodes  (achieved  as  of  2012) 

Security 

AES  256  for  user  and  control  plane 
traffic 

At  multiple  levels,  Type  1,  MIILS 

Modulation  (downlink) 

OFDM  (QPSK,  16QAM,  64QAM...) 

OFDM  (similar) 

Modulation  (uplink) 

SC-FDMA 

OFDM 

2x1  MIMO 

Mandatory 

Not  supported 

2x2  MIMO 

Not  supported 

Mandatory 

Not  supported 

4x4  MIMO 

Not  supported 

Mandatory 

Not  supported 

SOURCES:  Andrei  Traian,  "An  Overview  of  Long  Term  Evolution  Advanced,"  Saint  Louis,  Mo.,  undated, 
Motorola,  "Long  Term  Evolution  (LTE):  A  Technical  Overview,"  Technical  White  Paper,  2007;  Alcatel 
Lucent,  "The  LTE  Network  Architecture:  A  Comprehensive  Tutorial,"  Strategic  White  Paper,  2009.  Joint 
Tactical  Radio  System  Operational  Requirements  Document,  revision  2.3,  2002;  JTRS  NED  SAR,  "Selected 
Acquisition  Report,"  Department  of  Defense,  December  31,  2011;  Government  Accountability  Office, 
2008. 

a  Applicable  to  WNW  and  MANETs  only. 

cess  multiple  input  and  output  signals).  JTRS  GMR  and  WNW  do  not  have  MIMO 
capability. 

Table  1.1  indicates  that  LTE  performance  is  superior  to  that  of  WNW  in  sev¬ 
eral  important  areas.  First,  LTE  channel  bandwidths  are  up  to  four  times  larger  than 
those  available  for  WNW.  This  means  that  LTE  can  use  more  spectrum  (if  it  is  avail- 
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able)  than  WNW.  Also,  LTE  peak  user  data  rates  are  higher  than  those  available  with 
WNW,  especially  in  the  downlink.  This  is  not  surprising  because  LTE  eNode  B  anten¬ 
nas  are  fixed  and  can  provide  more  gain  and  power  than  a  GMR  vehicle-mounted 
radio.  More  important,  the  performance  difference  between  LTE  and  WNW  is  much 
larger  for  large  networks.  The  peak  data  rates  shown  are  characteristic  of  a  two-node 
network.  For  networks  of  larger  sizes,  the  average  data  rate  to  a  single  node  will  be 
smaller  than  the  peak  data  rate.  The  average  data  rates  provided  in  these  networks  is 
a  function  of  their  spectral  efficiency.  The  peak  data  rates  shown  in  Table  1.1  require 
some  explanation.  The  WNW  peak  data  rate  shown  is  for  the  entire  WNW  wireless 
channel,  which  includes  the  channel  bandwidth  needed  for  the  control  plane  (which 
allocates  modulation,  power,  and  bandwidth  settings  that  each  node  should  use  in  the 
shared  channel).  Field  tests  of  GMR  and  WNW  reveal  that  the  control  plane  takes 
up  progressively  more  bandwidth  as  the  number  of  nodes  in  the  WNW  network  is 
increased  and  even  more  when  a  significant  fraction  of  the  WNW  nodes  are  mobile. 
This  is  because  WNW,  in  contrast  to  LTE,  is  an  infrastructureless  network  design, 
so  when  network  resources  have  to  be  changed  because  of  network  topology  changes, 
WNW  nodes  have  to  negotiate  and  reallocate  network  resources  (e.g.,  time  slots)  to 
try  and  optimize  network  performance.  On  the  other  hand,  the  LTE  RAN  is  centrally 
controlled  by  the  eNode  B.  In  the  LTE  RAN,  much  less  negotiation  is  required,  which 
limits  LTE  control  plane  traffic.  With  a  channel  bandwidth  of  20  MHz,  LTE  can 
achieve  a  spectral  efficiency  of  over  6  bits  per  Hertz,  which  is  much  higher  than  the 
spectral  efficiency  of  WNW  observed  in  field  tests.34 

It  is  unlikely  that  the  greater  security  capabilities  of  WNW  cannot  explain  all  the 
differences  in  performance  highlighted  in  Table  1.1,  especially  for  user  data  rates  and 
network  sizes.  The  latter  is  especially  important,  as  LTE  wireless  networks  can  scale  up 
in  size  to  support  over  1,000  nodes  (UEs)  from  a  single  eNode  B.  Some  of  these  dif¬ 
ferences  are  probably  due  to  the  more  advanced  technologies  used  in  LTE  that  are  not 
available  in  WNW  (e.g.,  MIMO  antenna  processing),  in  addition  to  the  performance 
penalties  incurred  by  WNW  as  a  result  of  its  substantial  security  requirements.  How¬ 
ever,  another  important  difference — maximum  supported  network  size — is  due  to  the 
more  demanding  operational  environment  that  WNW  nodes  are  designed  to  operate 
in.  WNW  supports  full  MANET  operation,  where  all  nodes  in  the  network  can  be 
mobile.  In  contrast,  in  an  LTE  network,  only  UE  nodes  are  designed  to  be  mobile.  The 
eNode  B  nodes  are  designed  to  remain  stationary,  which  greatly  simplifies  aspects  of 
the  network  design  and  network  management. 

Another  aspect  of  the  two  network  designs  that  is  also  substantially  different 
(and  not  alluded  to  in  Table  1.1)  is  network  control  functions.  In  the  LTE  network 
architecture,  network  control  functions  are  built  into  specific  nodes,  described  in  the 
section  above,  that  reside  in  the  backhaul  network  and  in  the  eNode  B.  UE  mobile 


34  In  this  unclassified  publication,  we  cannot  provide  an  estimate  of  WNW  spectral  efficiency. 
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handsets  have  minimal  network  intelligence  and  control  capabilities  built  into  them. 
This  enables  mobile  handsets  to  be  smaller,  to  consume  less  power,  and  to  be  less  costly. 

In  contrast,  the  WNW  network  design  is  entirely  different.  WNW  network  and 
control  capabilities  are  built  into  each  node.  This  eliminates  the  need  for  one  or  more 
control  nodes  in  a  WNW  network  but  significantly  increases  the  intelligence  and  pro¬ 
cessing  power  requirements  for  GMR  WNW  terminals.  This  in  turn  resulted  in  the 
larger  size,  processing  power,  and  prime  electric  power  requirements  for  the  GMR. 
Although  the  WNW  network  design  enables  peer-to-peer  networking  and  robust  and 
survivable  network  design,  it  raises  the  cost  of  each  network  node  considerably. 

Comparison  of  Development  Time  Lines 

Both  LTE  and  JTRS  WNW  are  the  products  of  extensive  requirements  processes  that 
incorporate  high-level  guidance  and  priorities  from  key  stakeholders.  First,  we  review 
the  development  time  line  for  JTRS  and  WNW. 

JTRS  and  WNW 

The  JTRS  program  was  originally  started  in  response  to  high-level  strategy.  In  August 
1997,  the  Joint  Requirements  Oversight  Council  (JROC)  approved  a  mission  needs 
statement  for  a  JTRS.  Six  months  later,  in  March  1998,  it  approved  the  operational 
requirements  document  for  a  joint  tactical  radio  program.  The  DoD  structured  this 
acquisition  program  to  fulfill  goals  established  by  Joint  Vision  2010. 35  Joint  Vision 
2010  posited  that,  because  of  an  increasingly  lethal  battlespace  caused  by  technology 
advances,  a  prudent  posturing  strategy  should  include  the  ability  for  “greater  mobility 
and  increased  dispersion”  of  military  personnel;  this  tactical  structure  requires  “addi¬ 
tional  communications  and  coordination  capabilities”  to  synchronize  these  dispersed 
elements.36  At  the  time,  no  proven  technical  solution  existed.  Acquisition  officials 
determined  that  a  new  JTRS  program  would  serve  as  the  technical  solution  to  these 
broad  military  goals. 

The  JTRS  program  suffered  requirements  perturbations  early  in  its  development. 
Some  requirements  changes  likely  resulted  from  interactions  between  industry  and 
program  officials,  as  program  officials  sought  to  determine  technology  limits.  The 
Army  held  an  Industry  Day  on  June  27,  2000,  with  the  stated  purpose  “to  discuss  the 
overall  strategy  for  and  structure  of  the  Army’s  program  from  the  present  development 
phase  to  the  production  phase  of  the  JTRS.” 

Figure  1.12  shows  that  the  JTRS  requirements  development  process  spanned 
about  six  years.  The  ORD  for  the  JTRS  program  went  through  several  iterations  from 
March  1998  to  April  2003,  when  Version  3.2  was  published.37  In  the  interim,  the 


33  Chairman,  Joint  Chiefs  of  Staff,  Joint  Vision  2010,  Department  of  Defense,  undated. 

36  Chairman,  Joint  Chiefs  of  Staff,  undated. 

37  See  Bob  Arguero,  “Army  to  Hold  Joint  Tactical  Radio  System  Industry  Day,”  GovCon,  June  20,  2000. 
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program  office  released  a  request  for  proposal  (RFP)  in  October  2001,  four  years  after 
the  release  of  a  related  mission  needs  statement  and  a  year  and  a  half  before  the  final 
ORD.  In  January  2002,  the  DoD  completed  the  JTRS  Analysis  of  Alternatives  (AoA). 
In  May  2002,  the  DoD  awarded  a  contract  to  Boeing  for  delivery  of  JTRS  Cluster  1 
(which  would  later  be  called  JTRS  GMR).  JTRS  Cluster  1  included  both  the  hardware 
and  software  for  the  radio  under  a  single  contract.  The  Army  served  as  program  man¬ 
ager  for  this  contract. 

In  2005  the  Cluster  1  program  encountered  a  number  problems.  These  problems 
included  size  and  weight  issues  (the  radio  was  too  big),  interoperability  concerns,  as  well 
as  issues  associated  with  its  security  architecture  (it  used  software-based  encryption).38 
These  issues  prompted  a  major  restructuring  of  the  program  in  March  2006,  splitting 
the  hardware  (now  called  GMR)  and  software  (SRW/WNW  waveforms)  into  two 
distinct  programs:  JTRS  GMR  and  JTRS  NED,  each  subject  to  its  own  testing  and 
schedules. 

Also  shown  in  Figure  1.12  are  key  milestones  in  the  development  ofWNW.  The 
first  WNW  field  demonstration  took  place  in  May  2006  when  four  GMRs  formed  a 
network  and  collectively  transmitted  data  at  a  rate  of  120  kbps.  Waveform  develop¬ 
ment  proceeded  over  the  next  five  years  or  so  until  WNW  was  tested  in  a  network 
of  30  nodes  in  2009.  In  late  2009,  WNW  was  ported  to  the  final  version  of  GMR 
hardware,  the  so-called  engineering  development  model  (EDM)  version  of  GMR.  The 
EDM  GMR  was  tested  in  networks  of  up  to  30  nodes  in  2009  and  later  at  the  GMR 
system  integration  test  (SIT)  in  September  2010.  The  program  was  never  able  to  dem¬ 
onstrate  a  working  network  of  70  to  120  nodes,  which  was  the  preferred  range  of  net¬ 
work  sizes  needed  to  network  all  senior  leaders  of  an  Army  BCT. 

The  JTRS  GMR  suffered  cost  growth  in  2008  and  again  in  2011,  ostensibly 
because  of  a  reduction  in  the  total  number  of  radio  units  needed  for  the  entire  Army 
force  after  the  cancelation  of  the  FCS  program.  Previous  RAND  research  on  the  root 
cause  of  JTRS  GMR  program  cost  growth,  indicates  that  the  reduction  in  units  likely 
occurred  because  of  system  capability  issues  and  not  because  of  exogenous  factors.39 
It  is  important  to  note  that  the  cost  growth  seen  in  the  GMR  program  was  not  mir¬ 
rored  in  the  NED  waveform  program.  Nevertheless,  WNW  scalability  limits  as  well 
as  GMR  size,  weight,  and  power  issues  prompted  the  reductions  in  the  quantity  of 
radios  purchased  and  subsequent  unit  cost  growth.  In  October  2011,  DoD  cancelled 
the  JTRS  GMR  program. 

In  summary,  the  JTRS  requirement  development  process  took  approximately  six 
years.  The  actual  development  of  the  JTRS  GMR  and  WNW  took  approximately  nine 


38  Andrew  Feickert,  “The  Joint  Tactical  Radio  System  (JTRS)  and  the  Army’s  Future  Combat  System  (FCS): 
Issues  for  Congress,”  DTIC  Document,  November  17,  2005. 

^  The  RAND  report  is  not  available  to  the  general  public. 
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years,  from  2002  to  2011.  The  development  of  GMR  and  perhaps  WNW  were  stopped 
before  it  was  shown  they  could  meet  their  program  requirements. 

LTE  and  SAE 

The  bottom  of  Figure  1.12  provides  a  time  line  for  the  LTE  development  process  that 
began  in  1999.  The  reader  will  recall  that  3GPP  was  formed  in  1998  and  initially 
was  given  the  task  of  creating  the  UMTS  standard  for  3G  cellular  networks.  Shortly 
after  that  standard  was  released  in  1999,  3GPP  was  given  the  new  task  by  its  leaders 
to  develop  a  new  standard  for  4G  wireless  cellular  networks  that  improved  on  the  3G 
standard  and  that  could  provide  greater  data  communications  capacity  for  network 
users. 

LTE  also  required  a  significant  amount  of  time  for  requirements  development. 
This  took  about  five  years  starting  in  1999  and  concluded  in  2004.  But  in  reality,  some 
requirements  have  continued  to  change  for  the  LTE  system  as  system  development 
proceeded  and  as  test  data  became  available.  Early  LTE  test  results  influenced  system 
design  and  improvements  and  have  resulted  in  the  evolution  of  LTE  requirements  over 
time.  This  was  especially  true  in  the  development  of  the  planned  Internet  protocol- 
based  voice  capability  for  LTE. 

LTE  development  was  initiated  formally  in  2004.  The  LTE  wireless  user  uplink 
and  downlink  waveforms  were  selected  in  2005.  The  first  version  of  LTE  was  labeled 
Release  8  and  was  completed  by  the  end  of 2006.  However,  one  can  see  from  the  figure 
that  a  considerable  amount  of  time  was  devoted  for  testing  of  Release  8 — from  mid- 
2005  to  late  2008.  It  was  not  until  late  2008  that  Release  8  was  stable  enough  to  be 
used  in  the  first  trial  networks. 

The  first  field  tests  of  LTE  Release  8  took  place  in  2009.  At  about  the  same  time 
Release  9  was  frozen,  which  was  the  first  version  of  LTE  that  has  been  deployed  on 
a  widespread  basis  in  commercial  networks.  The  first  commercial  deployments  took 
place  in  mid-2010.  By  2011,  LTE  Release  9  was  deployed  on  a  large  scale  in  all  major 
regions  of  the  world. 

One  can  see  from  the  time  line  in  Figure  1.1  that  initial  LTE  requirements  devel¬ 
opment  took  about  six  years  to  complete,  from  1999  to  2004.  But  one  should  be  aware 
that  LTE  requirements  development  was  not  fully  completed  then  and  future  versions 
or  releases  of  the  LTE  standard  are  planned.  It  is  important  to  note  that  it  will  not  be 
until  Release  10  that  a  fully  integrated  LTE  network  solution  will  be  provided.  Release 
10  will  provide  both  integrated  voice  and  data  communications  capabilities  in  the  same 
IP-based  backhaul  network.  LTE  Release  8  and  9  do  not  provide  such  a  capability.  In 
current  LTE  networks,  voice  communications  are  provided  by  the  3G  network. 
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Cost  Comparison 

JTRS  Development  Costs 

In  attempting  to  compare  the  development  strategies  of  LTE  and  WNW,  a  necessary 
task  is  comparing  total  costs.  Although  ideal  cost  comparisons  would  isolate  hard¬ 
ware  and  software  development,  as  well  as  program  management  costs,  as  illustrated 
above,  the  original  iteration  of  the  JTRS  program  integrated  both  the  hardware  and 
waveforms  into  one  program,  to  facilitate  management  of  program  interdependencies. 
Available  data  provide  only  a  rough  cost  division  of  both  efforts.  Further,  the  restruc¬ 
ture  in  2006  that  separated  both  programs  possibly  shifted  faults  of  WNW  develop¬ 
ment  into  GMR  costs  through  unit  reductions.  Thus,  the  most  accurate  cost  for  the 
development  of  the  JTRS  WNW  capability  is  combined  software  (WNW/SRW)  and 
hardware  (GMR)  RDT&E  costs  as  reported  by  the  DoD. 

Over  the  course  of  the  program,  DoD  invested  $3.2  billion  in  the  development 
of  JTRS  GMR  and  NED,  resulting  in  a  canceled  GMR  and  a  functioning  WNW 
waveform.  As  discussed  elsewhere  in  this  chapter,  the  dollar  value  invested  in  JTRS 
RDT&E  is  much  smaller  than  the  amount  invested  in  the  creation  of  the  fully  func¬ 
tioning  LTE  waveform.  However,  the  obvious  question  is  whether  the  dollar  value 
invested  is  the  driving  factor  for  the  JTRS  failure  or  whether  other  consequential 
differences  resulted  from  the  government/industry  partnership  nonexistent  in  self- 
interested,  private-market  collaboration.  A  fundamental  question  arises  about  the  abil¬ 
ity  of  a  government  acquisition  program  to  fund  and  adequately  incentivize  single  con¬ 
tractor  development  of  unproven  technology,  regardless  of  cost. 

RAND’s  previous  work  found  that  as  the  JTRS  GMR  program  devolved,  pro¬ 
gram  subcontractors  proffered  novel,  low-cost,  partial  substitute  radio  capability  out¬ 
side  the  prime  contractor  chain  of  command.40  The  existence  of  these  low-cost  alter¬ 
natives  contributed  to  the  demise  of  JTRS.  Currently,  the  question  remains  whether 
this  outcome  was  structurally  inevitable,  a  result  of  a  single  buyer — the  government — 
attempting  to  pay  a  single  contractor — Boeing — for  an  untested  capability  to  be  deliv¬ 
ered  nearly  a  decade  after  the  initial  request. 

We  used  cost  estimate  data  reported  by  the  program  manager  to  the  Defense 
Acquisition  Management  Information  Retrieval  (DAMIR)  system  on  a  yearly  basis  in 
the  SARs.  Given  the  perturbations  of  the  program  structure,  where  the  software  and 
hardware  were  alternatively  part  of  a  single  program  then  part  of  separate  but  highly 
dependent  programs,  we  chose  to  present  cost  estimates  for  the  combined  GMR-related 
software  and  hardware  programs  that  directly  pertained  to  WNW  development.  Con¬ 
sequently,  we  combine  the  RDT&E  costs  of  the  GMR  and  JTRS  NED  programs  and 
estimate  that  the  total  RDT&E  cost  for  the  development  of  JTRS  GMR  and  WNW 
is  $3.2  billion. 
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Harris  Corporation  offered  the  PRC-117G  in  2011. 
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The  numbers  in  Figure  1.13  provide  data  on  how  much  the  program  estimated 
it  spent  per  year;  it  does  not  illustrate  a  differential  between  how  much  the  program 
thought  it  would  spend  in  any  given  year  earlier  in  the  program.  In  fact,  in  2002,  at 
the  JTRS  program  baseline,  cost  estimators  placed  RDT&E  costs  for  the  JTRS  hard¬ 
ware  at  $845.1  million.  By  2011,  the  estimate  for  total  RDT&E  costs  for  hardware 
had  nearly  doubled,  to  $1.5  billion  (revised  to  2002  comparable  dollars).  Figure  1.14 
illustrates  this  manifestation  of  cost  changes.  (One  reason  for  program  cancelation 
included  the  significant  cost  increases  beyond  initial  estimation.) 

As  the  total  program  cost  estimates  grew  for  both  the  hardware  and  software, 
managers  sought  to  control  expenses  by  removing  originally  planned  efforts  from  the 
program.  In  2006,  DoD  removed  two  particularly  difficult  planned  hardware  instan¬ 
tiations  from  the  program:  the  Army  Aviation  Rotary  Wing  (AARW)  and  Air  Force 
Tactical  Air  Control  Party  (TACP).  The  sudden  drop  in  2006  spending,  seen  in  Figure 
1.13,  reflects  this  program  restructure;  however,  total  cost  increases  were  not  stemmed 
by  this  move.  In  the  original  plan,  JTRS  program  would  develop  33  software  defined 
waveforms  (32  legacy  waveforms  and  WNW),  whereas  the  final  program  included 
only  8. 41 


Figure  1.13 

JTRS  NED  and  GMR  RDT&E  Expenditures  (Estimate  1) 


SOURCES:  JTRS  GMR  SAR  12/31/2012.  JTRS  NED  SAR  09/30/3011. 

NOTE:  Data  show  actual  expenditures  through  201 1  and  projected  expenditures  after  that. 
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41  General  Accounting  Office,  Challenges  and  Risks  Associated  with  the  Joint  Tactical  Radio  System  Program — 
Briefing  to  the  House  Committee  on  Appropriations  Subcommittee  on  Defense,  May  5,  2003. 
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Figure  1.14 

JTRS  NED  and  GMR  RDT&E  Estimated  Total  Expenditures  Over  Time  (Estimate  2) 


3,500 


2002  2002  2003  2004  2005  2007  2007  2007  2007  2009  2010  2011 

SOURCES:  JTRS  GMR  SAR  2002-2011;  JTRS  NED  SAR  2002-2011. 
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LTE  Development  Costs 

To  obtain  a  point  of  comparison  for  JTRS/WNW  development  cost,  we  estimated 
how  much  RDT&E  funding  was  invested  globally  into  developing  LTE. 

Approach 

Since  it  was  impossible  to  obtain  exact  data  on  corporate  expenditures  for  LTE  devel¬ 
opment,  we  developed  an  approach  that  enabled  us  to  make  a  rough  estimate  for  this 
key  metric  based  on  publicly  available  information. 

We  first  identified  the  companies  involved  in  the  core  activities  related  to  LTE 
development.  We  then  assessed  their  product  slate  and  their  sales  figures  and  analyzed 
their  patent  portfolios.  This  allowed  us  to  estimate  the  percentage  of  each  company’s 
business  that  was  LTE-related. 

We  then  researched  corporate  records  to  obtain  annual  RDT&E  budgets  for  these 
companies,  which  in  combination  with  the  LTE-related  share  of  business  allowed  us  to 
estimate  each  company’s  LTE-related  RDT&E  expenditures  (Figure  1.15). 

Step  1:  Companies  Involved  in  LTE  Development 

Most  companies  involved  in  LTE  development  were  members  of  the  “LTE/SAETrial 
Initiative”  (LSTI)  that  was  active  from  May  2007  to  January  2011. 42  LSTI  was  an  “open 
initiative”  of  equipment  vendors  and  operators  and  its  objective  was  to  “drive  indus¬ 
trialization  of  3GPP  LTE/SAE  technology,  demonstrate  LTE/SAE  capabilities  against 
[  .  .  .  ]  requirements,  [and]  stimulate  development  of  the  LTE/SAE  ecosystem.”43 

LSTI  was  founded  by  Alcatel-Lucent,  Ericsson,  Nokia,  Nokia  Siemens  Networks, 
Nortel,  Orange,  T-Mobile  (Deutsche  Telekom),  and  Vodafone.  By  2009,  when  the 


42  “LSTI  Job  Complete,”  3GPP  homepage,  2011. 

43  “An  Update  from  the  LTE/SAE  Trial  Initiative,”  slide  presentation,  January  26,  2009. 


42  DoD  and  Commercial  Advanced  Waveform  Developments  and  Programs  with  Nunn-McCurdy  Breaches 


Figure  1.15 

Process  for  Estimating  Industry  Expenditures  for  LTE  Development 


Identify  companies  involved 
in  LTE  development 


Estimate  LTE-related  RDT&E  expenditures: 
$LTE  =  $  RDT&E  *  %LTE 
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first  commercial  deployment  of  an  LTE  network  took  place,44  LSTI  had  grown  to  32 
members,45  which  were  active  in  business  areas  including  component  manufacturing, 
infrastructure  development,  services,  testing,  handset  manufacturing,  and  commercial 
mobile  service  providers  (CMSPs).  Our  analysis  focuses  on  these  members,  listed  in 
Table  1.2.  Note  that  Nortel  bled  for  bankruptcy  in  2009,  and  T-Mobile  is  owned  by 
German  company  Deutsche  Telekom.  Ericsson  was  involved  in  handset  manufactur¬ 
ing  through  Sony  Ericsson,  from  2001  to  2012. 

Steps  2  and  3:  Share  of  Business  Related  to  LTE 

Most  of  the  companies  listed  in  Table  1.2  have  multiple  lines  of  business,  and  some  are 
global  conglomerates  dealing  in  a  variety  of  goods  and  services.  To  estimate  the  share 
of  each  company’s  business  that  is  actually  related  to  LTE,  we  looked  at  product  slates 
and  associated  sales  figures  where  available,  which  we  obtained  from  business  intel¬ 
ligence  databases.  We  also  analyzed  patents  bled  by  each  company  to  determine  what 
share  of  patents — and  thus,  by  extension,  of  development  work — was  related  to  LTE. 


44  Phil  Goldstein,  “TeliaSonera  Launches  First  Commercial  LTE  Network,”  FierceWireless,  December  14,  2009. 

45  Julius  Robson,  “The  LTE/SAE  Trial  Initiative:  Taking  LTE/SAE  from  Specification  to  Rollout,”  IEEE  Com- 
munications  Magazine,  April  2009. 
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Table  1.2 
Members  of  LSTI 


LSTI 

Member 

Sector 

Company 

Country 

2007 

2009 

Compo¬ 

nents 

Infra¬ 

structure 

Services 

Testing 

Hand¬ 

sets 

CMSP 

Alcatel-Lucent 

France 

X 

X 

X 

X 

Ericsson 

Sweden 

X 

X 

X 

X 

Nokia  &  Nokia 
Siemens  Networks 

Finland 

X 

X 

X 

X 

X 

Nortel 

Canada 

X 

X 

X 

Orange  (France 
Telecom) 

France 

X 

X 

X 

T-Mobile 

(Deutsche  Telekom) 

United  States/ 
Germany 

X 

X 

X 

Vodafone 

United  Kingdom 

X 

X 

X 

China  Mobile 

China  (PRC) 

X 

X 

Huawei 

China  (PRC) 

X 

X 

X 

Samsung 

South  Korea 

X 

X 

X 

Agilent 

United  States 

X 

X 

Azimuth  Systems 

United  States 

X 

X 

Freescale 

United  States 

X 

X 

Fujitsu 

Japan 

X 

X 

LG  Electronics 

South  Korea 

X 

X 

Motorola 

United  States 

X 

X 

X 

X 

NEC 

Japan 

X 

X 

X 

X 

Nethawk  (EXFO) 

Canada 

X 

X 

NTT  DoCoMo 

Japan 

X 

X 

Panasonic 

Japan 

X 

X 

Qualcomm 

United  States 

X 

X 

Rohde  &  Schwarz 

Germany 

X 

X 

SFR 

France 

X 

X 

Signalion 

Germany 

X 

X 

SK  Telecom 

South  Korea 

X 

X 

Starent  Networks 

United  States 

X 

X 

ST-NXP 

Switzerland 

X 

X 

Tektronix 

United  States 

X 

X 

Telecom  Italia 

Italy 

X 

X 

Telefonica 

Spain 

X 

X 

ZT 

China  (PRC) 

X 

X 
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Product  Slates  and  Sales  Figures 

The  primary  source  for  each  company’s  product  slate  and  sales  figures  was  “Hoover’s 
Company  Records,”  which  we  accessed  through  RAND’s  subscription  to  the  “Lexis- 
Nexis  Company  Dossier”  service.  Data  for  some  companies  that  were  not  covered  in 
Hoover’s  were  obtained  from  other  sources:  the  “Corporate  Affiliations”  database  for 
ZT,  “OneSource”  for  Starent,  and  “Creditreform”  for  Signalion.  These  databases  were 
also  accessed  through  Lexis-Nexis.  Product  slates  for  SFR  and  France  Telecom  Orange 
were  determined  from  company  websites.  The  databases  and  websites  were  queried  in 
late  April  and  early  May  2012,  and  the  results  reflect  the  status  as  of  that  time.  Figure 
1.16  shows  an  example  of  an  entry  from  Hoover’s,  including  sales  figures  and  a  listing 
of  the  business  categories  in  which  the  company  is  active. 

We  used  sales  figures,  where  available,  to  approximate  each  company’s  share  of 
wireless-related  business  and  the  business  categories  that  it  was  active  in.  We  used  the 
Hoover’s  database  (and  other  corporate  business  activity  databases  for  specific  compa¬ 
nies  that  could  not  be  found  in  Hoover’s)  to  estimate  the  percentage  of  the  individual 
companies’  business  that  related  to  wireless  telecom.  If  such  information  was  not  avail¬ 
able  in  Hoover’s  or  the  other  corporate  databases,  we  used  50  percent.  For  most  of  the 
large  players  in  the  business  wireless  telecom  business,  the  required  information  was 
available  in  Hoover’s.  Such  data  were  not  available  for  a  few  firms,  mainly  smaller  for¬ 
eign  companies.  In  that  case,  we  used  50  percent  to  estimate  (1)  the  percentage  of  their 
business  focused  on  wireless  telecom  and  (2)  the  percentage  of  their  wireless  R&D 
funds  used  for  wireless  telecom  R&D.  For  these  relatively  small  firms,  we  believe  that 
these  estimates  are  reasonable  because  even  though  they  are  small  firms,  they  are  still 
members  of  3GPP.  It  requires  substantial  resources  to  gain  entry  and  participate  in  the 
consortium.  In  addition,  member  companies  are  expected  to  provide  technical  experts 
for  working  committees  that  meet  several  times  a  year  in  various  locations  around  the 
world.  A  small  firm  that  offered  only  a  few  products  in  the  wireless  telecom  space  and 
which  did  most  of  its  business  in  other  market  categories  would  be  unlikely  to  become 
a  member  of  the  consortium.  We  realize  that  there  is  some  error  with  the  50  percent 
figure  for  a  few  3GPP  firms,  but  we  believe  that  it  represents  a  reasonable  lower  bound 
for  the  R&D  costs  of  smaller  firms  that  participate  in  the  wireless  telecom  industry. 
The  resulting  percentages  are  listed  in  Table  1.3. 

Patent  Analysis 

To  obtain  a  second  estimate  for  the  share  of  a  company’s  business  devoted  to  LTE, 
we  examined  the  share  of  LTE-related  patents  that  each  company  filed  during  the 
time  period  under  analysis,  using  data  from  ETSI’s  Intellectual  Property  Rights  (IPR) 
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Figure  1.16 

Screenshot  of  Company  Entry  in  Hoover's 
Business  Intelligence  Database 
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Table  1.3 

LTE-Related  RDT&E  Expenditures  for  LSTI  Members  (in  U.S.  $  Millions) 


Based  on  Based  on  Based  on 

Product  Slate  Total  Patents  Original  Patents 


RDT&E 

Spending 

LTE 

LTE 

LTE 

LTE 

LTE 

LTE 

Company 

Country 

2005-2011 

Share 

rdt8ie 

Share 

rdt8ie 

Share 

RDT&E 

Alcatel-Lucent 

France 

22,086 

20% 

4,420 

39% 

8,610 

47% 

10,380 

Ericsson 

Sweden 

29,555 

30% 

8,870 

65% 

19,210 

66% 

19,510 

Nokia  &  Nokia 
Siemens  Networks 

Finland 

49,882 

20% 

9,980 

35% 

17,290 

33% 

16,290 

Nortel 

Canada 

4,106 

17% 

700 

15% 

620 

0% 

0 

Orange 

France 

7,952 

25% 

1,990 

0% 

0 

0% 

0 

T-Mobile 

(Deutsche  Telekom) 

United  States/ 
Germany 

4,285 

50% 

2,140 

31% 

1,330 

0% 

0 

Vodafone 

United  Kingdom 

2,908 

44% 

1,280 

0% 

0 

0% 

0 

China  Mobile 

China  (PRC) 

k 

50% 

* 

0% 

k 

0% 

k 

Huawei 

China  (PRC) 

9,870 

14% 

1,350 

4% 

370 

1% 

70 

Samsung 

South  Korea 

28,736 

13% 

3,660 

14% 

4,110 

42% 

11,930 

Agilent 

United  States 

2,607 

6% 

160 

0% 

0 

0% 

0 

Azimuth  Systems 

United  States 

k 

6% 

k 

0% 

* 

0% 

k 

Freescale 

United  States 

3,552 

8% 

300 

0% 

0 

0% 

0 

Fujitsu 

Japan 

10,994 

4% 

470 

0% 

0 

0% 

0 

LG  Electronics 

South  Korea 

5,250 

16% 

840 

33% 

1710 

39% 

2,040 

Motorola 

United  States 

6,954 

25% 

1,740 

0% 

0 

0% 

0 

NEC 

Japan 

13,303 

10% 

1,260 

30% 

3,960 

47% 

6,280 

Nethawk  (EXFO) 

Canada 

141 

13% 

20 

0% 

0 

0% 

0 

NTT  DoCoMo 

Japan 

5,064 

50% 

2,530 

40% 

2,000 

64% 

3,250 

Panasonic 

Japan 

24,413 

13% 

3,050 

55% 

13,350 

56% 

13,700 

Qualcomm 

United  States 

10,722 

25% 

2,680 

11% 

1,210 

11% 

1,170 

Rohde  &  Schwarz 

Germany 

1312 

17% 

220 

0% 

0 

0% 

0 

SFR 

France 

369 

25% 

90 

0% 

0 

0% 

0 

Signalion 

Germany 

★ 

50% 

★ 

0% 

* 

0% 

* 

SK  Telecom 

South  Korea 

882 

42% 

370 

0% 

0 

0% 

0 

Starent  Networks 

United  States 

16,304 

50% 

8,150 

0% 

0 

0% 

0 

ST-NXP 

Switzerland 

7,386 

19% 

1,380 

0% 

0 

0% 

0 

Tektronix 

United  States 

1,048 

5% 

50 

0% 

0 

0% 

0 

Telecom  Italia 

Italy 

3,912 

10% 

390 

0% 

0 

0% 

0 

Telefonica 

Spain 

4,264 

5% 

210 

0% 

0 

0% 

0 

ZT 

China  (PRC) 

* 

13% 

* 

0% 

k 

0% 

* 

Total 

277,857 

58,300 

73,770 

84,620 

NOTE:  An  asterisk  indicates  that  no  data  were  available. 
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database.46  This  database  contains  a  public  record  of  all  patents  or  other  expressions  of 
IPR  reported  or  declared  to  ETSI,  including  IPR  declared  with  patent  offices  from  92 
different  countries.  An  example  entry  from  the  database  and  its  associated  information 
is  shown  in  Table  1.4. 

For  each  LSTI  member  firm,  we  calculated  the  LTE-related  share  of  total  patents 
and  original  patents  issued  from  2005  to  2011.  An  LTE  “total  patent”  refers  to  a  patent 
that  was  originally  declared  for  the  3GPP  Release  5  standard  and  declared  for  LTE  at  a 
later  date.  In  comparison,  an  LTE  “original  patent”  refers  to  a  patent  where  the  declar¬ 
ing  company  indicated  at  the  start  that  LTE  was  the  primary  applicable  standard  for 
the  patent.  As  one  can  tell,  most  patents  in  this  field  have  applications  to  many  differ¬ 
ent  communications  standards,  and,  as  a  result,  companies  list  all  the  different  stan¬ 
dards  applicable  for  each  patent. 

For  the  purposes  of  our  calculations,  the  LTE-related  share  of  total  patents  and 
original  patents  issued  from  2005  to  2011  served  as  a  proxy  for  the  share  of  a  com¬ 
pany’s  RDT&E  expenditures  associated  with  LTE.  The  resulting  percentages  are  listed 
in  Table  1.3  as  well. 

Step  4:  Corporate  RDT&E  Budgets 

Most  LSTI  member  companies  are  publicly  traded  and  thus  published  annual  financial 
reports  during  the  period  under  analysis.  If  these  companies  traded  on  U.S.  exchanges, 
the  data  were  crosschecked  using  their  annual  Form  10-K  Securities  and  Exchange 
Commission  (SEC)  filings.47  These  reports  included  each  company’s  RDT&E  spend¬ 
ing.  Most  of  these  reports  provided  figures  in  U.S.  dollars;  for  those  that  did  not,  the 
amounts  were  converted  into  U.S.  dollars  using  the  exchange  rates  for  the  end  of  the 
respective  year.  Note  that  the  amounts  are  not  adjusted  for  inflation. 


Table  1.4 

Example  ETSI  Database  Entry 


Patents 

Publication 

Title 

Patent  Offices 

Declarations 

US20080007516 

US2009179755  A1 

Overload  control 
method 

US  (United  States) 

ISLD-200911-002 

Declaration 

Date 

Patent  Families 

Companies 

Standards 

ETSI  Projects 

10/27/2009 

|US20080007516 

ALCATEL-LUCENT 

TS  136  423  V8.7.0 

LTE 

46  ETSI  is  an  independent,  nonprofit  organization  that  is  recognized  by  the  European  Union  as  a  European 
Standards  Organization.  The  ETSI  IPR  database  contains  data  on  IPRs  “which  have  been  notified  to  ETSI  as 
being  essential,  or  potentially  essential,  to  ETSI  Standards  and  Technical  Specifications.”  See  “ETSI  IPR  Data¬ 
base,”  ETSI  homepage,  2011. 

47  Both  the  Annual  Reports  (financial  performance)  and  the  Form  10-K  annual  filings  to  the  SEC  are  available 
at  each  company’s  respective  website.  Form  10-K  annual  filings  are  also  available  on  the  SEC  EDGAR  database. 
(See  “EDGAR  Database,”  U.S.  Securities  and  Exchange  Commission,  2012.  Copies  of  these  reports  and  filings 
are  also  kept  on  file  at  RAND. 
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RDT&E  expenditures  were  recorded  for  2005  to  2011  for  founding  members 
of  LSTI  and  for  2008  to  2011  for  members  that  joined  later.  The  results  are  listed  in 
Table  1.5. 

Four  companies — Chinese  firms  Huawei  and  ZT,  Azimuth  Systems  from  the 
United  States,  and  the  German  company  Signalion — are  not  publicly  traded  and  thus 
no  budget  information  was  available  for  them,  and  none  is  included  in  Table  1.5.  How¬ 
ever,  at  least  some  of  these  companies  can  be  expected  to  have  invested  in  RDT&E; 
thus,  the  grand  total  given  in  the  table  does  not  include  any  RDT&E  for  these  four 
companies  and  is,  therefore,  a  conservative  estimate. 

Step  5:  Results 

Multiplying  the  percentages  determined  in  Step  3  with  the  overall  RDT&E  budgets 
for  each  company  determined  in  Step  4  yields  three  different  estimates  for  the  LTE- 
related  RDT&E  expenditures  of  the  companies  included  in  this  analysis,  ranging  from 
approximately  $58  billion  to  $85  billion  (U.S.  dollars),  as  indicated  in  Table  1.3.  Figure 
1.17  shows  the  distribution  of  estimated  LTE  development  cost  across  the  LSTI  mem¬ 
bers,  ranked  from  highest  to  lowest  contributor,  and  excluding  China  Mobile,  Azi¬ 
muth  Systems,  Signalion,  and  ZT  since,  as  mentioned  above,  no  data  were  available 
for  these  companies.  The  figure  also  shows  the  estimated  percentage  of  LTE-related 
RDT&E,  based  on  the  product  slate  approach,  for  each  company. 

Accounting  for  Competition  and  Redundancy 

It  could  be  argued  that  industry  invested  more  in  LTE  development  than  would  have 
been  necessary,  since,  as  shown  in  Table  1.6,  multiple  companies  cover  each  business 
sector,  and  this  redundancy  led  to  parallel  development  efforts  and  thus  increased 
expenditures.  To  take  this  into  account  and  to  allow  for  a  more  accurate  comparison 
with  the  cost  ofWNW  development,  we  developed  a  “redundancy  factor”  by  which  to 
adjust  the  estimated  cost. 

Using  the  allocation  of  companies  to  business  sectors  provided  in  Table  1.2,  we 
counted  how  many  companies  were  active  in  each  sector  and  then  divided  this  number 
by  a  desired  redundancy  value.  We  set  this  value  at  four,  indicating  that  four  competi¬ 
tors  in  a  given  sector  are  the  minimum  number  required  to  foster  healthy  competition 
and  guard  against  the  creation  of  monopolies  through  mergers.  We  used  four  as  the 
minimum  number  of  firms  in  a  market  to  define  the  redundancy  factor  in  our  analysis 
because  it  relates  directly  to  a  key  metric  used  in  antitrust  analysis  from  the  econom¬ 
ics  literature.  This  is  the  command,  control,  computers,  and  communications  (C4) 
factor,  which  measures  market  concentration  or  power  of  the  four  largest  firms  in  a 
market.  For  example,  a  C4  of  100  percent  indicates  monopoly  conditions.  A  C4  over  60 
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Table  1.5 

Corporate  RDT&E  Budgets  for  Period  of  Interest 

[sTi 

Member _ RDT&E  Expenditures  (U.S.  $  Millions) 


Total 

2005- 


Company 

Country 

2007 

2009 

2005 

2006 

2007 

2008 

2009 

2010 

2011 

2011 

Alcatel-Lucent 

France 

X 

X 

1,537 

1,934 

4,348 

3,886 

3,731 

3,449 

3,201 

22,086 

Ericsson 

Sweden 

X 

X 

3,077 

4,012 

4,498 

3,979 

4,604 

4,658 

4,727 

29,555 

Nokia  &  Nokia 
Siemens  Networks 

Finland 

X 

X 

4,530 

5,141 

8,295 

8,412 

8,469 

7,769 

7,266 

49,882 

Nortel 

Canada 

X 

X 

* 

* 

1,723 

1,573 

703 

107 

0 

4,106 

Orange 

France 

X 

X 

848 

1,129 

1,316 

1,269 

1,235 

1,120 

1,035 

7,952 

T-Mobile 

(Deutsche  Telekom) 

United  States/ 
Germany 

X 

X 

592 

660 

856 

826 

628 

408 

315 

4,285 

Vodafone 

United  Kingdom 

X 

X 

341 

403 

467 

339 

446 

469 

443 

2,908 

China  Mobile 

China  (PRC) 

X 

* 

* 

* 

* 

* 

Huawei 

China  (PRC) 

X 

1,527 

1,951 

2,669 

3,723 

9,870 

Samsung 

South  Korea 

X 

5,609 

6,485 

7,989 

8,653 

28,736 

Agilent 

United  States 

X 

704 

642 

612 

649 

2,607 

Azimuth  Systems 

United  States 

X 

* 

* 

* 

•k 

* 

Freescale 

United  States 

X 

1,140 

833 

782 

797 

3,552 

Fujitsu 

Japan 

X 

2,764 

2,438 

2,896 

2,896 

10,994 

LG  Electronics 

South  Korea 

X 

1,374 

1,089 

1,323 

1,464 

5,250 

Motorola 

United  States 

X 

2,358 

1,591 

1,479 

1,526 

6,954 

NEC 

Japan 

X 

3,896 

3,756 

3,384 

2,267 

13,303 

Nethawk  (EXFO) 

Canada 

X 

27 

28 

38 

48 

141 

NTT  DoCoMo 

Japan 

X 

1,216 

1,092 

1,347 

1,409 

5,064 

Panasonic 

Japan 

X 

6,135 

5,614 

5,847 

6,817 

24,413 

Qualcomm 

United  States 

X 

2,281 

2,995 

2,451 

2,995 

10,722 

Rohde  &  Schwarz 

Germany 

X 

339 

343 

319 

312 

1312 

SFR 

France 

X 

89 

100 

93 

87 

369 

Signalion 

Germany 

X 

* 

* 

* 

* 

* 

SK  Telecom 

South  Korea 

X 

190 

207 

234 

251 

882 

Starent  Networks 

United  States 

X 

0 

5,208 

5,273 

5,823 

16,304 

ST-NXP 

Switzerland 

X 

319 

2,365 

2,350 

2,352 

7,386 

Tektronix 

United  States 

X 

262 

216 

258 

312 

1,048 

Telecom  Italia 

Italy 

X 

989 

1,207 

925 

791 

3,912 

Telefonica 

Spain 

X 

942 

993 

1,056 

1,273 

4,264 

ZT 

China  (PRC) 

X 

* 

* 

* 

* 

* 

Total 

7 

31 

10,925 

13,279 

21,503 

52,445 

58,969 

59,305 

61,432 

277,857 

NOTE:  An  asterisk  indicates  that  no  data  were  available. 


50  DoD  and  Commercial  Advanced  Waveform  Developments  and  Programs  with  Nunn-McCurdy  Breaches 


Figure  1.17 

Estimated  LTE  Development  Cost  and  LTE  Share  of  RDT&E  Investments  by  Company 
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NOTE:  No  financial  data  were  available  for  China  Mobile3,  Azimuth  Systems,  Signalion,  and  ZTE. 
3  Permanent  and  elected  members  of  LSTI's  Steering  Board  for  2009. 
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percent  indicates  oligopoly.  We  realize  that  economists  use  other,  more  detailed  met¬ 
rics  to  measure  market  concentration  such  as  the  Hirschman-Herfindahl  Index,  but 
these  other  metrics  would  require  much  more  analysis  to  apply  in  our  case.48  The  C4 
metric,  on  the  other  hand,  can  be  applied  directly  to  our  data  set.  In  the  text,  we  have 
added  a  discussion  of  this  issue  and  a  citation  that  provides  an  explanation  of  the  C4 
metric.  Table  1.6  shows  the  resulting  redundancy  factor  values  for  each  sector. 


Table  1.6 

Redundancy  Factors  for  Each  Business  Sector 


Business  Sector 

Components 

Infrastructure 

Services 

Testing 

Handsets 

CMSP 

LSTI  members  active  in  sector 

8 

8 

3 

6 

7 

9 

Desired  number  of  competitors 

4 

Redundancy  factor  for  sector 

2.0 

2.0 

1.0 

1.5 

1.8 

2.3 

48  Janice  Hauge  and  Mark  Jamison,  Analyzing  Telecommunications  Market  Competition:  Foundations  for  Best 
Practices ,  Gainesville,  Fla.:  Public  Utility  Research  Center,  University  of  Florida,  October  29,  2009. 
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The  redundancy  factors  for  each  sector  were  averaged,  resulting  in  an  overall 
redundancy  factor  of  1.75  by  which  to  divide  the  cost  estimates  provided  above.  Using 
this  approach,  the  idealized  LTE-related  RDT&E  investment  cost  to  be  used  for  com¬ 
parison  with  WNW  development  cost  ranges  from  $33  billion  to  $48  billion  (U.S. 
dollars). 

We  also  performed  an  additional  estimation  to  bound  our  estimate,  using 
5  percent  as  an  arbitrary  lower  limit  for  the  average  LTE-related  share  of  a  compa¬ 
ny’s  RDT&E  budget,  which  resulted  in  total  LTE-related  RDT&E  expenditures  of 
approximately  $14  billion  (U.S.  dollars). 

These  estimates  indicate  that  industry  invested  significantly  in  LTE  development 
over  the  period  2005  to  2011. 

Caveats 

The  estimation  process  we  developed  relies  on  several  assumptions  that  limit  the  accu¬ 
racy  of  the  results.  As  is  evident  from  the  descriptions  above,  our  estimations  for  the 
LTE-related  share  of  a  company’s  business  are  rough  approximations.  Furthermore, 
a  company’s  product  slate  and  involvement  in  LTE  development  may  have  changed 
over  the  years,  but  we  had  only  2012  Hoover’s  data  available.  In  particular,  Nortel’s 
bankruptcy  in  2009  influenced  that  company’s  activities.  For  some  companies,  only 
limited  data  were  available,  and  no  RDT&E  investment  figures  were  available  for  four 
privately  held  companies.  Finally,  we  did  not  attempt  to  capture  the  expenses  of  com¬ 
panies  that  were  not  members  of  LSTI  but  may  have  made  contributions  to  LTE  devel¬ 
opment  nevertheless. 

Summary 

The  results  of  the  cost  analysis  presented  in  this  chapter  are  summarized  below.  We 
remind  the  reader  that  we  did  not  have  access  to  original  source  data  from  LTE  devel¬ 
opers  because  private  industry  firms  typically  do  not  break  out  how  they  spend  R&D 
funds.  For  that  reason,  we  had  to  estimate  LTE  development  costs  using  publicly  avail¬ 
able  information.  On  the  other  hand,  because  LTE  development  was  led  by  3GPP  and 
because  3GPP  identified  the  firms  that  participated  in  LTE  development,  we  know 
who  these  private  firms  are. 

We  developed  LTE  R&D  estimates  using  three  approaches:  (1)  by  estimating  the 
R&D  costs  for  LTE  as  a  percentage  of  the  total  R&D  costs  for  all  products  produced 
by  each  firm,  (2)  by  estimating  LTE  development  costs  on  the  basis  of  original  patents 
issued  to  individual  firms  for  LTE  intellectual  products,  and  (3)  by  estimating  LTE 
development  costs  on  the  basis  of  the  total  patents  issued  to  each  firm  involved  in  the 
development  of  LTE.  The  results  of  these  three  different  estimating  techniques  are 
shown  in  Table  1 .7. 

We  also  estimated  LTE  RDT&E  costs  using  the  product  slate-based  approach  but 
using  the  very  conservative  assumption  that  only  5  percent  of  the  publicly  announced 
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Table  1.7 

Total  LTE/SAE  Costs  (2007-2011) 

Estimation  Approach 

LTE  RDT&E 
Expenditures 
(U.S.  $  billions) 

Product  slate-based 

58 

Optimistic  (5%) 

14 

Original  patent  shares 

85 

Total  patent  shares 

85 

RDT&E  funds  each  firm  expended  were  devoted  to  LTE.  We  characterize  this  as  the 
“optimistic  (5%)”  cost  estimate  approach.  Using  this  approach,  we  estimate  that  total 
worldwide  LTE  RDT&E  costs  were  approximately  $14  billion  (U.S.  dollars). 

It  was  much  easier  to  estimate  GMR  and  WNW  RDT&E  costs  using  cost  data 
from  JTRS  program  offices.  Those  results  are  given  in  Table  1.8,  which  shows  that  the 
total  GMR  and  WNW  RDT&E  costs  were  approximately  $3.2  billion  (U.S.  dollars). 

The  results  shown  in  Tables  1.7  and  1.8  indicate  that  LTE/SAE  RDT&E  costs  are 
substantially  greater  than  those  for  GMR  and  WNW,  regardless  of  the  method  chosen 
to  estimate  these  costs. 

LTE  RDT&E  costs  were  a  factor  of  4  to  24  times  greater  than  those  for  GMR 
and  WNW,  which  raises  the  question  as  to  whether  the  DoD  did  not  expend  enough 
funds  on  R&D  to  reduce  the  technical  risks  associated  with  JTRS  GMR  and  WNW 
development. 


Conclusions 

This  study  examined  the  technical  performance  capabilities,  development  costs,  and 
organizations  that  led  to  the  development  of  the  LTE  and  JTRS  GMR  WNW  com¬ 
munication  networking  systems.  Below,  we  summarize  the  major  findings  of  this  study. 


Table  1.8 

Total  GMR  &  WNW  Costs  (2002-2011) 

Program 

GMR  and  WNW  RDT&E 
Expenditures 
(U.S.  $  billions) 

GMR 

1.5 

JTRS  NED 

1.7 

Total 

3.2 
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Findings 

When  comparing  the  JTRS  GMR  WNW  program  with  LTE,  four  issues  stand  out. 
First,  the  amount  spent  on  JTRS  GMR  and  WNW  RDT&E  appears  to  be  far  less 
than  was  needed  when  compared  to  the  amount  spent  for  RDT&E  by  commercial  firms 
in  the  development  of  LTE.  Second,  the  more  evolutionary  development  approach  for 
LTE  along  with  the  higher  RDT&E  expenditure  enabled  project  personnel  to  deal 
with  system  performance  and  technology  risk  issues  as  they  came  up.  Third,  the  JTRS 
program  structure  that  separated  hardware  and  software  development  coupled  with  a 
“big  bang”  acquisition  approach  (the  delivery  of  a  system  that  could  meet  all  require¬ 
ments  at  Milestone  C)  not  only  complicated  coordination  needs  among  programs  but 
also  may  have  delayed  the  discovery  of  integration  problems  until  it  was  too  late  or  too 
near  Milestone  C.  And  last,  the  IP  model  used  in  the  DoD  JTRS  program  may  have 
prevented  the  incorporation  of  needed  technologies  into  the  program  that  could  have 
reduced  technical  risks  and  performance  issues.  In  contrast,  the  more  inclusive  3GPP 
IP  model  encouraged  the  incorporation  of  a  worldwide  and  world-class  set  of  technol¬ 
ogy  IP  into  LTE.  We  elaborate  on  these  issues  below. 

Organizations  and  Intellectual  Property  Rights 

Although  the  two  systems  differ  significantly  in  technical  and  design  aspects,  they 
share  common  underlying  wireless  communications  technologies. 

The  organizational  structures  that  spearheaded  the  development  of  these  two 
systems  are  quite  different.  Furthermore,  the  LTE  and  GMR  WNW  development 
approaches  and  architecture  are  significantly  different.  The  LTE  architecture  is  not 
software-defined,  and  hardware  components  (microchips  in  particular)  are  shared 
between  collaborating  vendors  but  software  developed  by  competing  commercial  ven¬ 
dors  is  not.  IP  rights  to  LTE  software  are  retained  by  individual  vendors  in  most  cases. 
In  comparison,  JTRS  GMR-WNW  architecture  is  software-defined:  The  software  is 
government-owned  and  can  be  shared  between  vendors,  if  approved  by  the  govern¬ 
ment,  whereas  GMR  hardware  is  not  shared  outside  the  original  development  team  of 
contractors. 

The  organizational  model  and  technology-sharing  rules  differ  substantially  in  the 
two  approaches.  JTRS  GMR  WNW  was  developed  in  a  traditional  DoD  acquisi¬ 
tion  program  with  IP  access  controlled  by  a  select  number  of  program  contractors 
and  government  personnel.  IP  for  executable  software  code  is  retained  by  the  govern¬ 
ment,  whereas  IP  for  hardware  designs  is  retained  by  individual  contractors.  The  GMR 
WNW  system  was  developed  by  two  acquisition  programs,  one  responsible  predomi¬ 
nantly  for  hardware  development  and  for  the  OS  software  for  the  hardware  platform 
(the  GMR  program)  and  other  responsible  predominantly  for  only  software  for  WNW 
(the  JTRS  NED  program).  One  complication  of  this  arrangement  is  that  a  significant 
amount  of  coordination  was  required  between  the  two  programs  for  software  develop¬ 
ment.  The  JTRS  JPEO  was  responsible  for  coordinating  and  aligning  products  of  the 


54  DoD  and  Commercial  Advanced  Waveform  Developments  and  Programs  with  Nunn-McCurdy  Breaches 


GMR  and  NED  programs.  One  mechanism  used  to  effect  the  coordination  of  soft¬ 
ware  development  efforts  was  the  development  of  a  set  of  software  standards  for  JTRS 
program  radios — the  SCA. 

In  comparison,  the  LTE/SAE  architecture  is  specified  by  a  set  of  technical  stan¬ 
dards  that  was  developed  by  the  3GPP  international  consortium.  The  consortium’s 
governing  body  is  a  global  set  of  regional  telecommunications  standards-setting  orga¬ 
nizations.  3GPP  also  includes  industry  firms.  LTE/SAE  standards  are  developed  by 
industry  experts  from  member  firms.  Individual  firms  have  the  option  to  offer  their 
own  IP  to  form  a  part  of  one  or  more  LTE/SAE  standards.  If  accepted  by  the  group, 
this  IP  is  then  available  to  other  3GPP  members  to  use  in  their  own  products  under  the 
terms  of  the  3GPP  essential  patent-licensing  policy.  IPR  are  retained  by  the  individual 
3GPP  member  firms,  but  licenses  for  essential  LTE/SAE  patents  must  be  made  avail¬ 
able  to  other  3GPP  member  firms  on  a  fair,  reasonable,  and  nondiscriminatory  basis. 

Development  Costs 

One  of  the  most  surprising  results  of  this  study  concerns  RDT&E  costs.  Significantly 
more  funds  were  expended  for  the  development  of  LTE  than  was  allocated  by  the  DoD 
for  the  development  of  JTRS  GMR  WNW.  We  note  that  some  of  the  difference  may 
stem  from  redundant  investments  by  industry  competitors,  but  we  tried  to  take  into 
account  this  effect  in  our  cost  analysis  methodology.  LTE  RDT&E  costs  were  at  least  a 
factor  of  four  greater  than  those  for  GMR  and  WN  and  were  more  likely  to  be  a  factor 
of  10  to  24  times  greater. 

This  finding  raises  the  question  as  to  whether  the  DoD  did  not  make  a  large 
enough  R&D  investment  to  reduce  the  substantial  technical  risks  associated  with  the 
advanced  technologies  needed  for  this  program. 

Of  course,  JTRS  is  not  the  only  DoD  program  that  has  suffered  significant  cost 
growth  over  the  past  decade.  And  perhaps  what  has  been  observed  with  JTRS  is  just 
symptomatic  of  a  larger  issue  within  the  DoD,  where  cost  estimation  processes  lead 
to  overly  optimistic  and  ultimately  unrealistic  cost  estimates  for  DoD  acquisition  pro¬ 
grams  that  require  the  development  of  advanced  technologies. 

Development  Time  Lines  and  Approaches 

The  development  time  lines  overall  for  JTRS  GMR  WNW  and  for  LTE  are  compa¬ 
rable.  Initial  requirements  development  for  the  GMR  program  took  approximately  six 
years  compared  to  about  five  years  for  LTE.  On  the  other  hand,  system  development 
took  nine  years  for  JTRS  GMR  and  WNW  and  only  six  years  for  LTE.  One  reason 
for  the  significant  difference  in  the  latter  time  line  is  that  there  was  more  concurrent 
development  of  requirements  and  systems  in  the  LTE  program. 

WNW  and  LTE  share  common  technologies  in  radio  frequency  modulation 
schemes,  but  LTE  incorporates  some  advanced  technologies  not  available  in  WNW 
such  as  MIMO  and  use  of  multiple  antennas  in  handsets  and  eNode  B  radio  access 
network  nodes.  LTE  also  employs  a  new  power-efficient,  high-performance,  an  uplink 
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waveform  and  wider  bandwidth  downlink  channels  than  are  available  in  WNW.  The 
LTE  network  architecture  has  demonstrated  that  it  is  scalable  and  can  support  over 
1,000  nodes  in  a  single  wireless  network. 

LTE  and  WNW  network  and  security  architectures  are  substantially  different. 
JTRS  WNW  is  a  single-tier,  peer-to-peer  network  where  each  node  is  identical  and  has 
all  the  necessary  intelligence  built  into  it  so  that  it  can  function  effectively  even  on  the 
move  as  part  of  a  MANET.  As  held  tests  have  shown,  WNW  network  scalability  has 
not  been  demonstrated  beyond  30  nodes.  In  contrast,  the  LTE/SAE  network  architec¬ 
ture  is  based  on  a  two-tier  network  where  a  single  radio  access  point  (eNode  B)  can 
support  over  1,000  wireless  nodes.  Finally,  both  network  architectures  have  yet  to  fully 
integrate  voice  and  data  communications  (although  3GPP  is  developing  plans  and 
standards  for  this  integration  in  future  releases  of  the  LTE  set  of  standards). 

The  development  approaches  for  the  two  initiatives  also  differ  significantly. 
The  JTRS  program  approach  initially  called  for  the  development  of  an  entirely  new 
MANET  communications  architecture  that  relied  on  a  number  of  new  and  unproven 
technologies.  In  other  words,  it  used  a  “big  bang”  type  of  an  approach  where  it  was 
assumed  that  an  entirely  new  system  could  be  developed  and  that  the  new  technolo¬ 
gies  needed  to  make  the  system  work  could  be  developed  concurrently  with  the  system 
itself.  Later,  after  the  program  had  encountered  significant  cost,  performance,  and 
schedule  problems,  the  program’s  organizational  approach  was  changed  significantly. 
In  the  restructured  program,  hardware  and  software  development  were  separated  into 
separate  programs.  It  is  debatable  whether  this  new  organizational  approach  was  a 
success. 

On  the  other  hand,  the  LTE  development  approach  was  decidedly  different.  As  its 
name  suggests,  private  industry  and  3GPP  took  an  evolutionary  approach  toward  the 
development  of  this  new  system  architecture.  LTE  represents  an  incremental  develop¬ 
ment  approach  for  next-generation  wireless  cellular  communications  networks.  Just  as 
in  current  3G  wireless  networks,  voice  and  data  backhaul  networks  are  separated  in  the 
initial  versions  of  LTE.  It  was  judged  to  be  too  difficult  to  integrate  them  along  with 
all  the  other  architectural  changes  that  were  made  in  the  initial  versions  of  the  LTE 
network  architecture.  In  these  initial  versions,  development  emphasis  was  placed  on 
improving  the  spectral  efficiency  and  performance  of  the  radio  access  network,  on  con¬ 
solidating  the  number  of  network  nodes  needed  in  the  backhaul  network,  and  finally 
on  transitioning  the  backhaul  network  to  a  standard  IP-based  network.  And  perhaps 
most  important,  the  fundamental  two-tiered  structure  of  current  wireless  cellular  net¬ 
works  was  preserved  in  LTE.  In  this  two-tiered  structure,  the  connection  between 
the  two  LTE  network  tiers  is  provided  by  the  eNode  B  network  node,  which  remains 
fixed.  This  is  in  contrast  to  the  MANET  network  designs  that  were  developed  as  part 
of  the  JTRS  GMR  and  WNW  programs,  in  which  no  network  nodes  have  to  remain 
stationary.  The  evolutionary  approach  taken  in  the  LTE  development  enabled  the  LTE 
network  to  scale  up  to  thousands  of  wireless  nodes;  in  contrast,  the  MANET  design 
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developed  for  GMR  and  WNW  number  was  never  able  to  scale  up  to  such  large  net¬ 
work  sizes.  The  technology  needed  to  do  this  remains  to  be  developed. 

Program  Outcomes 

It  is  important  to  consider  the  outcomes  of  these  two  development  efforts.  The  JTRS 
GMR  WNW  development  effort  lasted  for  approximately  12  years  and  has  not  yet 
resulted  in  a  product  fielded  to  the  U.S.  Army.  The  GMR  program  was  canceled  in 
2011,  and  the  JTRS  program  responsible  for  the  development  of  WNW  was  restruc¬ 
tured  in  2012  and  is  now  called  the  JTNC.  The  JTNC  will  be  responsible  for  the  con¬ 
tinued  development  of  JTRS  waveforms  and  will  assume  the  responsibility  for  tran¬ 
sitioning  these  waveforms  (including  WNW)  to  commercial  radios  (radios  developed 
by  defense  contractors  using  their  own  internal  R&D  funds).  So  in  terms  of  hardware 
development  efforts,  the  JTRS  GMR  program  was  not  a  success  as  it  did  not  lead  to 
fielded  hardware  products.  However,  the  waveform  development  part  of  the  JTRS 
family  programs  (JTRS  NED)  was  somewhat  more  successful  and  led  to  the  develop¬ 
ment  of  advanced  networking  waveforms  that  may  be  used  in  future  commercial  radios 
whose  initial  versions  have  already  been  fielded  to  U.S.  Army  and  Marine  Corps  units. 

Recommendations 

Our  analysis  shows  that  in  hindsight,  GMR  and  WNW  development  costs  do  not 
appear  to  be  exorbitant  given  the  challenging  requirements  originally  established  for 
the  JTRS  program,  and  perhaps  the  DoD  did  not  invest  enough  in  upfront  R&D  to 
reduce  WNW  technical  risks.  If  WNW  is  to  be  scaled  up  to  larger  networks  of  100 
nodes  or  more,  additional  RDT&E  funds  will  be  necessary.  Such  an  R&D  program 
could  be  carried  out  by  JTNC,  or  by  the  Army’s  Research,  Development  and  Engi¬ 
neering  Command,  using  commercial  radios,  when  they  become  available. 

Another  possible  approach  will  be  necessary  if  the  Army  and  DoD  decide  that 
WNW  cannot  meet  future  warfighter  needs.  If  this  is  the  case,  given  the  limited  R&D 
dollars  likely  to  be  available  in  future  budgets,  the  DoD  should  consider  adapting 
LTE  to  meet  its  operational  and  security  needs.  In  this  case,  a  new  DoD  R&D  invest¬ 
ment  strategy  will  be  needed  that  leverages  best  practices  from  the  commercial  sector, 
including  the  use  of  technical  standards,  to  foster  collaboration  among  competing 
defense  contractors.  Such  an  approach  for  the  DoD  would  be  new  but  could  leverage 
the  model  established  by  3GPR 

In  any  case,  alternatives  to  the  JTRS  development  approach  based  on  software- 
defined  radios  and  waveforms  should  be  considered  in  future  DoD  communications 
systems  developments.  The  JTRS  SCA  is  aging  quickly  in  a  world  of  rapidly  changing 
software  architecture  standards  and  may  no  longer  provide  the  best  standards  plat¬ 
form  for  radio  development.  Furthermore,  it  is  not  consistent  with  recent  industry 
trends  toward  tighter  integration  of  software  and  hardware.  A  technical  standards- 
based  approach  that  also  incorporates  provisions  for  licensing  essential  patents  at  fair 
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and  reasonable  prices  and  in  a  nondiscriminating  fashion  should  be  core  tenets  of  this 
new  R&D  strategy. 

In  addition,  alternatives  to  the  standard  DoD  acquisition  process  should  be  sought 
out  for  DoD  IT  and  communications  system  programs.  DoD  IT  and  communications 
should  follow  the  LTE  development  model  and  pursue  an  evolutionary  approach.  Such 
an  approach  is  possible  if  essential  patents  for  current  and  future  (evolved)  systems 
can  be  licensed  in  a  fair  and  nondiscriminatory  basis  with  competing  contractors. 
The  DoD  should  also  initiate  such  programs  with  a  new  standards  development  phase 
that  enables  a  broad  cross-section  of  industry  to  contribute  and  share  patents  deemed 
essential  for  system  development.  The  DoD  should  also  consider  altering  later  program 
phases  to  enable  broader  industry  competition  that  such  an  IPR-sharing  approach  will 
enable. 

Finally,  the  DoD  should  cast  a  wider  technology  net  to  incorporate  the  latest  com¬ 
mercially  developed  advanced  information  technologies.  To  do  this,  the  DoD  should 
consider  joining  3GPP  and  explore  the  option  of  enabling  3GPP  common  essential 
patents  to  be  used  in  military  systems. 


CHAPTER  TWO 


Multiple  Nunn-McCurdy  Breaches 


The  Performance  Assessments  and  Root  Cause  Analysis  (PARCA)  office  asked  RAND 
to  review  programs  that  had  breached  Nunn-McCurdy  thresholds  with  the  main 
objective  of  understanding  how  these  program  breaches  evolved.  We  were  asked  the 
following  three  research  questions: 

•  Are  programs  that  have  a  Nunn-McCurdy  breach  more  likely  to  breach  again? 

•  What  can  we  learn  from  the  cost  growth  trends  of  those  programs  that  have  had 
multiple  Nunn-McCurdy  breaches? 

•  What  can  we  learn  from  the  management  actions  taken  on  programs  with  mul¬ 
tiple  breaches?  Are  there  common  issues  that  were  missed  at  the  first  breach? 


Analytical  Approach 

We  used  a  two-phased  analytic  approach  for  this  exploratory  research.  In  the  first 
phase  of  this  research,  we  addressed  the  first  two  questions.  We  tried  to  understand  the 
trends  in  unit  cost  growth  for  the  programs  with  multiple  breaches.  We  also  compared 
the  relative  numbers  of  programs  that  experience  single  breaches  with  those  that  had 
multiple  breaches.  In  the  second  phase,  we  explored  issues  of  programs  with  repeat 
Nunn-McCurdy  breaches,  focusing  on  what  was  missed  at  the  initial  breach  and  how 
management  actions  evolved.1  We  used  case  studies  of  four  programs  to  perform  this 
second  phase. 

Because  we  had  limited  resources  for  this  research  effort,  we  focused  on  a  readily 
available  data  source  for  our  research — the  SARs.  Although  there  are  limitations  in 
using  these  data,2  these  reports  contain  the  most  consistent  and  readily  available  data 
to  which  we  have  access  for  a  large  cross-section  of  programs.  For  the  first  phase  of  our 


1  A  Nunn-McCurdy  breach  occurs  when  the  procurement  acquisition  unit  cost  (PAUC)  or  average  procure¬ 
ment  unit  cost  (APUC)  exceeds  certain  percentages.  See  Appendix  A  for  fuller  explanations. 

2  See,  for  example,  Paul  G.  Hough,  Pitfalls  in  Calculating  Cost  Growth  from  Selected  Acquisition  Reports,  Santa 
Monica,  Calif.:  RAND  Corporation,  N-3136-AF,  1992. 
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research,  we  identified  programs  (using  SARs)  reporting  Nunn-McCurdy  breaches  in 
more  than  one  fiscal  year.  At  this  point,  we  did  not  sort  out  programs  that  may  not 
have  had  legitimate  repeat  breaches;  however,  we  were  able  to  plot  both  a  program’s 
breach  reporting  history  and  trends  in  PAUC  or  APUC.  In  addition  to  collecting 
data  on  the  programs,  we  also  collected  any  recent  analysis  of  programs  with  repeated 
Nunn-McCurdy  breaches  in  the  literature. 

The  data  we  used  came  from  the  Office  of  the  DAMIR  system,  which  allowed 
us  to  review  data  on  MDAPs  going  back  to  1997.  We  extracted  the  following  program 
details  from  SARs: 

•  name  (including  subprogram) 

•  acquisition  category  (ACAT)  status 

•  service  (component) 

•  milestone  information  (SAR  date  where  data  were  collected;  Milestone  II/B, 
Milestone  C/III,  and  IOC/FUE  [initial  operating  capability/first  unit  equipped] 
when  available) 

•  Nunn-McCurdy  unit  cost  metrics  provided  in  the  SARs  (PAUC  and  APUC) 
against  current  and  original  baselines.  This  includes  whether  the  breach  was  criti¬ 
cal  or  significant  and  the  exact  percentage  change  in  base  year  dollars  (BY  $).3 

Next,  we  separated  programs  that  reported  more  than  one  Nunn-McCurdy 
breach  during  the  timeframe  of  the  dataset  without  looking  for  any  programs  that 
might  have  reported  the  same  breach  over  multiple  fiscal  years.  After  identifying  these 
programs,  we  created  two  sets  of  plots  for  each  program:  breach  reporting  history  and 
trends  in  APUC  and  PAUC.  For  the  APUC  and  PAUC,  we  specifically  looked  at  the 
following  percentage  changes  over  time: 

•  PAUC  against  the  current  baseline  in  BY  $ 

•  APUC  against  the  current  baseline  in  BY  $ 

•  PAUC  against  the  original/revised  baseline  in  BY  $ 

•  APUC  against  the  original/revised  baseline  in  BY  $. 

Next,  we  extracted  the  explanations  that  were  provided  in  the  SARs  for  the 
reported  breaches  for  each  individual  program.  This  was  the  first  step  in  developing  an 
understanding  of  whether  reports  over  multiple  fiscal  years  for  a  particular  program 
could  be  considered  as  a  new  breach  or  just  a  continuation  of  the  original  breach. 

After  constructing  the  plots  for  programs  with  repeated  Nunn-McCurdy  breaches, 
we  also  constructed  tables  that  plotted  the  breach  history  for  each  program  over  time. 
These  tables  included  the  SAR  reporting  years  and  indicated  any  critical  or  significant 


3  A  breach  that  exceeds  the  current  budget  PAUC  or  APUC  by  15  percent  is  considered  significant,  and  one  that 
exceeds  the  current  budget  by  25  percent  is  considered  critical. 
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Nunn-McCurdy  breaches.  This  same  table  provided  data  on  whether  the  program  had 
a  change  in  its  acquisition  program  baseline  (APB)  during  that  same  reporting  period. 

In  the  second  phase  of  this  research,  we  conducted  an  exploratory  analysis  of 
actions  taken  during  breaches  using  SARs  and  Acquisition  Decision  Memoranda 
(ADMs)  with  a  focus  on  four  programs: 

•  Space-Based  Infrared  System  (SBIRS  High)  (satellite) 

•  C-130  Avionics  Modernization  Program  (C-130  AMP)  (aircraft  electronics) 

•  Joint  Primary  Aircraft  Training  System  (JPATS)  (aircraft  training) 

•  H-l  Upgrades  (4BW/4BN)  (aircraft). 

This  second  phase  attempted  to  examine  what  the  programs  fixed  and  did  not  fix 
after  the  previous  breaches.  We  then  focused  on  identifying  alternatives  as  to  why  a 
program  might  breach  multiple  times.  These  reasons  included  the  following: 

•  Something  new  and  adverse  happened. 

•  The  DoD  did  not  identify  the  problem  or  identified  the  wrong  problem  at  the 
previous  breach. 

•  The  DoD  identified  the  problem  that  caused  cost  growth  to  date  but  did  not  rec¬ 
ognize  how  the  problem  would  continue  to  affect  the  program  in  the  future. 

•  The  DoD  identified  the  problem  but  did  not  take  action  to  address  it. 

•  The  DoD  identified  the  problem  but  the  actions  taken  were  ineffective.  This  could 
be  because  the  actions  taken  were  insufficient  or  not  persistent. 

For  this  second  phase,  we  collected  and  reviewed  all  available  SARs  and  ADMs 
for  these  programs  and  focused  on  the  following  details  regarding  the  programs: 

•  overall  program  history 

•  whether  a  breach  occurred  during  each  SAR/ADM  reporting  period 

•  main  challenges  or  specific  problems  that  affected  the  program  (e.g.,  technology, 
cost,  schedule)  and  possibly  led  to  the  breaches 

•  actions  taken  before,  during,  and  after  breaches  using  SARs  and  ADMs 

•  implementation  of  solutions  for  problems  that  caused  the  breaches. 

Note  that  this  second  phase  is  highly  speculative.  Because  of  limits  on  study 
funding  and  the  desire  not  to  burden  programs,  we  drew  inferences  from  the  pro¬ 
gram  records  exclusively  from  the  SARs.  We  did  not  discuss  these  interpretations  with 
anyone  (either  contractor  or  government)  related  to  the  programs.  Thus,  we  could  have 
a  misinformed  view  as  a  result  of  the  limited  information  contained  within  the  SARs. 

The  remainder  of  this  chapter  is  broken  into  three  sections.  The  next  section 
reviews  the  first  phase  of  this  work:  the  major  changes  to  the  Nunn-McCurdy  legisla¬ 
tion  in  2006,  counting  program  breaches,  and  a  summary  of  the  multiple  breach  sta- 
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tistics.  The  section  following  that  provides  detailed  discussions  of  the  four  programs 
selected  for  the  second  phase  and  examines  actions  taken.  The  final  section  summa¬ 
rizes  our  observations  with  respect  to  the  three  research  questions. 


Phase  1:  Exploring  Multiple  Breaches 

Changes  to  the  Nunn-McCurdy  Legislation 

Since  the  1980s,  the  Nunn-McCurdy  legislation  has  undergone  several  changes.  How¬ 
ever,  one  significant  change  appeared  in  the  fiscal  year  (FY)  2006  National  Defense 
Authorization  Act  (P.L.  109-163):  Congress  directed  that  cost  growth  be  measured 
against  the  original  baseline  estimate  (in  addition  to  the  current  baseline).  As  a  result 
of  this  legislation,  more  programs  had  Nunn-McCurdy  breaches  than  otherwise  would 
have  been  the  case  under  the  original  act  because  programs  could  rebaseline  to  avoid 
breaches  before  this  legislation.  “According  to  DOD,  11  programs  that  did  not  have 
a  Nunn-McCurdy  breach  before  the  new  FY2006  requirements  were  recategorized  as 
having  significant  breaches  as  a  result  of  the  legislation’s  new  original  baseline.”4  Some 
of  the  repeat  Nunn-McCurdy  breaches  can  be  attributed  in  part  to  changes  in  legisla¬ 
tion  rather  than  new  program  issues. 

Counting  Nunn-McCurdy  Breaches 

Counting  the  number  of  breaches  for  a  program  that  has  multiple  breaches  is  not  as 
straightforward  as  it  might  seem  on  the  surface.  Several  important  factors  require  con¬ 
sideration  in  identifying  programs  with  multiple  breaches: 

•  Rebaselines  and  revised  APBs  can  distort  the  count  before  2006; 

•  Whether  transitioning  from  “significant”  to  “critical”  Nunn-McCurdy  breaches 
counts  as  separate  breaches5; 

•  A  program  might  report  a  breach  for  multiple  fiscal  years  until  it  is  restructured; 

•  Whether  the  PAUC  and  APUC  breaches  are  independent  or  related; 

•  The  change  in  Nunn-McCurdy  law  in  fiscal  year  2006,  i.e.,  the  additional  report¬ 
ing  requirement  that  required  unit  cost  to  be  reported  relative  to  the  program’s 
original  baseline6; 

•  How  to  count  programs  that  may  split  into  multiple  programs. 


4  Moshe  Schwartz,  The  Nunn-McCurdy  Act:  Background,  Analysis,  and  Issues  for  Congress,  Congressional 
Research  Service,  January  31,  2011,  p.  10. 

5  A  “significant”  breach  is  when  the  current  baseline  estimate  is  breached  by  15  percent  or  the  original  baseline 
estimate  is  breached  by  30  percent.  A  “critical”  breach  is  when  the  current  baseline  estimate  is  breached  by  25 
percent  or  the  original  baseline  estimate  is  breached  by  50  percent.  See  Appendix  A  for  further  discussion. 

6  The  requirement  to  report  breaches  still  exists,  and  the  thresholds  are  lower  than  in  the  original  baseline.  See 
Appendix  A  on  the  history  of  Nunn  McCurdy  legislation. 
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RAND  Methodology  for  Counting  Multiple  Breaches 

For  this  analysis,  we  tried  to  take  as  simple  an  approach  as  possible.  Once  a  program 
breaches  either  APUC  or  PAUC  (either  significant  or  critical),  that  counts  as  a  breach. 
We  did  not  consider  that  a  program  going  from  significant  to  critical  as  an  additional 
breach  but  rather  as  just  a  further  evolution  of  the  same  breach.  Likewise,  we  observed 
generally  that  APUC  and  PAUC  were  so  correlated  in  terms  of  breaches,  that  count¬ 
ing  breaches  of  APUC  and  PAUC  independently  did  not  make  sense.  Also,  if  one  or 
the  other  lags  by  a  year  or  so,  that  would  not  constitute  a  second  breach.  For  example, 
if  a  program  had  a  critical  breach  in  APUC  one  fiscal  year  and  critical  or  significant 
breach  in  PAUC  the  following  year,  we  did  not  count  that  as  two  breaches  but  rather 
as  a  single  breach.  The  only  time  we  restarted  the  counting  is  when  there  had  been  a 
rebaseline  (before  2006)  or  a  recertification. 

GAO  Methodology  for  Counting  Multiple  Breaches 

The  GAO  uses  a  slightly  different  methodology.  In  testimony  to  Congress  in  March 
2011,  Michael  J.  Sullivan  (Director  Acquisition  and  Sourcing  Management  at  the 
GAO)  briefly  explained  how  GAO  counted  programs  with  multiple  breaches: 

To  identify  trends  in  Nunn-McCurdy  breaches,  we  collected  and  analyzed  exist¬ 
ing  data  on  breaches  from  DOD’s  Defense  Acquisition  Management  Information 
Retrieval  system,  which  contains  data  on  breaches  since  1997.  DOD  officials  also 
provided  us  with  a  list  of  programs  that  breached  the  cost  growth  thresholds  since 
1997,  which  we  analyzed  to  remove  duplicate  entries.  In  addition,  we  reviewed 
analyses  by  the  Office  of  the  Director  of  Cost  Assessment  and  Program  Evalua¬ 
tion  to  verify  our  data.  We  utilized  information  from  SARs  for  individual  weapon 
systems  to  explore  trends  by  various  program  characteristics  including  military 
service,  type  of  weapon  system,  and  contractor.  To  identify  factors  responsible  for 
trends  in  Nunn-McCurdy  breaches,  we  reviewed  DOD’s  root  cause  analyses  and 
analyzed  data  from  SARs,  compared  breach  trends  to  statutory  changes,  and  sum¬ 
marized  our  past  findings  on  programs  that  have  experienced  breaches.  To  identify 
factors  responsible  for  trends  and  identify  changes  DOD  is  making  or  proposing 
to  make  to  the  Nunn-McCurdy  process,  we  interviewed  relevant  officials  from  the 
offices  of  the  Undersecretary  of  Defense  for  Acquisition,  Technology,  and  Logis¬ 
tics;  Performance  Assessments  and  Root  Cause  Analyses;  Cost  Assessment  and 
Program  Evaluation;  the  Comptroller;  and  the  Joint  Staff.  We  also  reviewed  DOD 
policy  memoranda  and  proposed  legislation  to  learn  about  the  current  policy  and 
proposed  legislative  changes.7 


7  Government  Accountability  Office,  The  Honorable  Thomas  Carper  Chairman  Committee  on  Homeland  Secu¬ 
rity  and  Governmental  Affairs  Subcommittee  on  Federal  Financial  Management,  Government  Information,  Federal 
Services,  and  International  Security  United  States  Senate  Subject:  Trends  in  Nunn-McCurdy  Cost  Breaches  for  Major 
Defense  Acquisition  Programs,  Washington,  D.C.,  GAO-11-295R,  March  9,  2011a,  pp.  1—2. 


64  DoD  and  Commercial  Advanced  Waveform  Developments  and  Programs  with  Nunn-McCurdy  Breaches 


In  addition,  the  GAO  considered  a  breach  as  a  duplicate  if  the  same  breach  was 
reported  in  multiple  SARs.  They  found  that  “significant”  breaches  were  the  most 
common  duplicates.8  Finally,  the  GAO  counted  breaches  from  multiple  subprograms 
within  one  major  program  as  one  breach  in  a  given  year;  and  if  there  were  both  critical 
and  significant  Nunn-McCurdy  breaches  in  the  same  SAR,  then  this  was  also  consid¬ 
ered  one  breach.9 

The  GAO  published  some  of  the  most  recent  work  done  on  multiple  breach  pro¬ 
grams.  In  March  2011,  it  released  its  report  on  Trends  in  Nunn-McCurdy  Breaches  and 
Tools  to  Manage  Weapon  Systems  Acquisition  Costs.  In  this  analysis  of  programs  with 
repeat  Nunn-McCurdy  breaches  for  PARCA,  we  used  the  GAO’s  count  as  a  basis  of 
comparison  with  our  own  count  of  the  same  major  weapon  system  programs  during 
the  same  time  period  (since  1997)  using  a  similar  source  (SARs). 

The  result  of  the  above  methodologies  for  counting  multiple  breach  programs 
appears  in  Table  2.1.  Each  methodology  identifies  18  multiple  breach  programs  since 
1997  through  SARs;  however,  the  number  of  breaches  varies  from  program  to  pro¬ 
gram.  In  only  five  of  the  18  programs  do  both  methodologies  agree  on  the  number  of 
multiple  breaches.  This  result  signifies  that  identifying  multiple  breach  programs  is  a 
challenging  task  that  is  subject  to  differences  in  interpretation.  Unfortunately,  we  are 
unable  to  explain  these  differences,  as  we  do  not  have  insight  into  the  precise  counting 
method  the  GAO  used. 

A  few  programs  merit  comment.  The  SSN  774  program  reached  a  significant 
breach  in  the  2002  SAR  but  was  later  rebaselined  for  the  2003  SAR,  so  that  breach 
went  away.  ITowever,  with  the  legislative  change  to  Nunn-McCurdy  procedures  in 
2006,  that  significant  breach  against  the  original  baseline  now  reappears.  We  counted 
this  as  a  single  breach  because  the  two  breaches  that  occurred  (and  are  reported  by 
GAO)  are  manifestations  of  the  same  original  growth.  Figure  2.1  shows  the  breach¬ 
reporting  history  for  the  SSN  774  program.  The  figure  displays  the  years  for  which  the 
SARs  report  a  breach,  either  critical  or  significant.  Significant  breaches  are  indicated 
by  an  “S”  with  a  yellow  background.  Critical  breaches  are  indicated  by  a  “C”  with  a 
red  background.  Breaches  are  indicated  against  both  baseline  criteria  (original  and  cur¬ 
rent).  Before  the  2005  SARs,  programs  were  not  required  to  report  against  the  original 
baseline;  hence,  this  is  greyed  out  in  the  figure.  Years  for  which  the  baseline  is  revised 
are  indicated  by  blue  shading. 

The  CFIEM  DEMIL  program  is  difficult  to  assess  because  it  had  two  compli¬ 
cations:  It  was  split  into  multiple  programs  and  its  breach(es)  transitioned  across  the 
2006  change  in  the  Nunn-McCurdy  legislation.  Because  of  these  issues,  it  is  difficult 
to  make  a  consistent  count  (hence  the  plus  signs  added  to  the  values,  which  could  be 
more  than  shown  in  the  table). 


8  Government  Accountability  Office,  2011a,  p.  1. 

9  Government  Accountability  Office,  2011a,  p.  6. 
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Table  2.1 

RAND  and  GAO  Counting  of  Multiple  Breach  Programs  (1997  Through  2009) 


Program 

Number  of  Nunn- 
McCurdy  Breaches 

RAND  GAO 

Advanced  extremely  high  frequency 

2 

3 

Armed  reconnaissance  helicopter 

1 

2 

Advanced  threat  infrared  countermeasures/common  missile  warning  system 

3 

3 

B-1B  conventional  missile  upgrade  program 

1 

2 

C-130  avionics  modernization  program 

2 

3 

Chemical  demilitarization  (Chem  Demil)/assembled  chemical  weapons 
alternatives 

2+ 

Chem  Demil/Chemical  Materials  Agency  (CMA) 

1  + 

3 

Chem  Demil/CMA  Newport 

1  + 

2 

Expeditionary  fighting  vehicle 

2 

2 

F-35 

2 

3 

Global  Hawk 

2 

3 

Guided  multiple  launch  rocket  system  (GMLRS)/GMLRS  alternative  warhead 

2 

HI  Upgrades 

2 

2 

Joint  air-to-surface  standoff  missile 

1 

2 

Javelin 

2 

2 

National  polar-orbiting  operational  environmental  satellite  system 

2 

2 

SBIRS  High 

2 

4 

Nuclear  submarine  (SSN)  774 

1 

2 

SOURCES:  RAND  analysis  of  SAR  data  from  1997  through  2009  and  Government  Accountability  Office, 
2011b,  p.  10. 

NOTES:  A  plus  sign  indicates  some  uncertainty  in  counting  the  number  of  breaches  for  the  Chem  Demil 
program  because  the  program  was  split  into  multiple  efforts  from  a  single  program.  The  count  could  be 
higher  if  breaches  for  the  presplit  program  are  included. 


Overall  Statistics  for  Multiple  Nunn-McCurdy  Breach  Programs 

Using  data  reported  in  both  annual  and  quarterly  SARs  in  DAMIR  from  1997  through 
June  2011  on  MDAPs,10  we  were  able  to  identify  individual  programs  with  one  or  more 
Nunn-McCurdy  breaches  as  described  above.  This  produced  a  sample  set  of  53  indi- 


10  We  excluded  programs  that  started  in  either  2010  or  2011.  These  programs  would  not  likely  experience  mul¬ 
tiple  breaches  because  of  their  limited  reporting. 
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Figure  2.1 

SSN  774  Nunn-McCurdy  Breach  History 
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vidual  programs  that  reported  at  least  one  Nunn-McCurdy  breach  from  1997  through 
June  2011.  We  also  gathered  a  complete  list  of  individual  programs  during  that  same 
time  period,  which  gave  us  the  total  set  of  MDAPs  at  130  programs  excluding  pro¬ 
grams  that  started  reporting  in  either  2010  or  2011.  We  identified  13  programs11  that 
have  had  multiple  or  repeat  Nunn-McCurdy  breaches.  Using  the  quantities  reported 
above,  Figure  2.2  shows  the  population  demographics  of  MDAPs  in  DAMIR  report¬ 
ing  unit  cost  measures  from  1997  to  June  2011.  Forty-one  percent  of  the  total  pro- 

Figure  2.2 

Populations  Demographics  (Based  on  DAMIR  Since  1997) 
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NOTE:  A  list  of  the  programs  appears  in  Appendix  B. 
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11  Our  dataset  that  characterizes  the  overall  statistics  on  Nunn-McCurdy  programs  expands  the  dataset  to 
include  programs  through  June  2011  whereas  the  programs  in  Table  2.1  ended  in  2009.  This  expansion  includes 
an  additional  multiple  breach  program:  JTRS  GMR. 
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grams  had  at  least  one  Nunn-McCurdy  breach,  whereas  only  10  percent  could  be  con¬ 
sidered  multiple  Nunn-McCurdy  breach  programs. 

The  data  in  Figure  2.2  suggest  that  it  is  not  the  case  that  programs  that  breach 
once  are  more  likely  to  breach  again.  About  40  percent  of  the  programs  breach  once, 
and  about  25  percent  of  the  programs  that  breach  once  have  further  breaches  How¬ 
ever,  we  must  caveat  this  statement  by  noting  that  the  sample  sizes  are  too  small  to  be 
conclusive.  Thus,  it  is  not  possible  to  conclude  that  programs  that  breach  are  likely  to 
breach  again. 


Phase  2:  Individual  Repeat  Nunn-McCurdy  Program  Examples 

We  chose  four  programs  for  which  to  examine  the  breach  history,  challenges,  and 
management  actions  in  more  depth.  These  programs  are  SBIRS  High,  C-130  AMP, 
JPATS,  and  H-l  Upgrades  (4BW/4BN).  Each  program,  with  the  exception  of  JPATS, 
was  a  multiple  breach  program  with  unique  issues  and  challenges.  JPATS  was  included 
with  the  other  programs  as  a  counter  example;  it  only  has  had  one  breach. 

SBIRS  High 

The  SBIRS  High  program  is  an  Air  Force  satellite  program.  According  to  the  DAMIR: 

SBIRS  High  is  an  integrated  system  consisting  of  multiple  space  and  ground  ele¬ 
ments,  with  incremental  deployment  phasing,  simultaneously  satisfying  require¬ 
ments  in  the  following  mission  areas:  Missile  Warning,  Missile  Defense,  Technical 
Intelligence  and  Battlespace  Awareness.  The  constellation  architecture  for  SBIRS 
High  includes  Highly  Elliptical  Orbit  (HEO)  sensors  and  Geosynchronous  Earth 
Orbit  (GEO)  satellites,  in  addition  to  the  following  ground  elements:  a  Continen¬ 
tal  United  States  (CONUS)-based  Mission  Control  Station  and  Mission  Control 
Station  Backup,  overseas  Relay  Ground  Stations,  Mobile  Ground  Stations,  and 
associated  communication  links.12 

The  program  was  awarded  its  Milestone  II  in  October  1996.  This  program  has 
the  most  extensive  Nunn-McCurdy  history  of  the  four  that  we  reviewed,  incurring 
breaches  in  four  of  13  years.  In  fact,  the  program  is  viewed  as  a  “repeat  offender”: 

Sen.  John  McCain  (R-AZ),  the  ranking  member  on  the  Senate  Armed  Services 
Committee,  railed  against  the  program’s  troubled  history  in  a  speech  last  Decem¬ 
ber  [2011].  “It  is  worth  bearing  in  mind  that  the  Government  Accountability 
Office’s  latest  March  9,  2011,  report  on  major  defense  acquisition  programs  notes 
that  SBIRS  has  the  odious  distinction  of  breaching  the  ‘Nunn-McCurdy’  law  on 


12  “SBIRS  HIGH  Selected  Acquisition  Report  (SAR),”  Defense  Acquisition  Management  Information  Retrieval 
(DAMIR),  December  31,  2010,  p.  3. 
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cost  growth  a  record  four  times — the  most  of  any  major  weapons  program.  It’s  a 
hall-of-famer,”  McCain  said.13 

However,  it  is  difficult  to  reconcile  the  program’s  status;  our  counting  scheme 
indicated  that  the  program  has  two  breaches.  However,  GAO  counts  four  breaches  for 
this  program  (see  Table  2.1).  Figure  2.3  illustrates  both  the  Nunn-McCurdy  breach 
history  and  the  program’s  new  or  revised  APBs.  SBIRS  High  had  critical  Nunn- 
McCurdy  breaches  in  three  years:  2001,  2002,  and  2005.  The  program  also  had  sig¬ 
nificant  Nunn-McCurdy  breaches  in  2004  and  2005;  but  there  have  been  no  reported 
Nunn-McCurdy  breaches  since  2005.  In  addition  to  the  breaches,  the  program  has 
also  had  a  series  of  revised  APBs.  After  its  Milestone  II  was  awarded  in  October  1996, 
the  program  subsequently  revised  its  APB  in  1998,  1999,  2003,  and  2006.  In  terms  of 
“counting”  the  number  of  breaches,  the  first  breach  occurred  in  2001  and  2002  and 
was  resolved  by  the  new  APB  in  2003.  The  second  breach  began  in  2004  and  ended  in 
2006  with  another,  new  APB. 

Figure  2.4  tracks  the  percentage  change  in  the  PAUC  and  APUC  since  Milestone 
B  was  awarded  in  1996  using  data  provided  in  various  annual  and  quarterly  SARs.  As 
can  be  seen  in  the  chart,  all  four  measures  of  unit  cost  grew  significantly  in  2002: 

•  The  PAUC  against  the  current  baseline  increased  99  percent. 

•  The  APUC  against  the  current  baseline  increased  69  percent. 

Although  not  as  pronounced  as  in  2002,  a  critical  breach  was  also  triggered  in 
2005  when  the  APUC  against  the  current  baseline  grew  33  percent.  It  is  interesting 

Figure  2.3 

SBIRS  High  Nunn-McCurdy  Breach  History 
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13  Christopher  J.  Castelli,  “Pentagon  Takes  Key  Step  Toward  Acquiring  New  SBIRS  Satellites,”  InsideDefense. 
com,  Inside  the  Pentagon,  March  15,  2012. 
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Figure  2.4 

SBIRS  High  PAUC  and  APUC  Trends 
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to  note  that  this  program  also  had  two  significant  declines  in  these  cost  growth  met¬ 
rics.  The  APUC  against  the  current  baseline  in  December  1999  fell  37  percent,  and 
in  December  2009,  the  PAUC  and  APUC  against  the  current  baselines  dropped  29 
percent  and  28  percent,  respectively. 

In  the  cost  variance  section,  the  SARs  provide  explanations  for  why  costs  have 
changed  from  the  prior  estimate  to  the  current  one.  These  differences  are  reported  as 
individual  reasons  grouped  by  appropriation  (e.g.,  RDT&E,  procurement,  military 
construction,  operations  and  support)  and  identified  with  a  specific  cost  variance  cat¬ 
egory.  There  are  seven  different  variance  categories.14  Bolten  et  al.15  describe  them  as 
follows:16 

1.  quantity:  cost  variance  resulting  from  a  change  in  the  number  of  end  items 
being  procured 


14  William  Past,  “Sources  of  Program  Cost  Growth,”  Defense  AT&L,  March-April  2007,  pp.  24-27. 

16  Joseph  G.  Bolten,,  Robert  S.  Leonard,  Mark  V.  Arena,  Obaid  Younossi,  and  Jerry  M.  Sollinger,  Sources  of 
Weapon  System  Cost  Growth:  Analysis  of  35  Major  Defense  Acquisition  Programs,  Santa  Monica,  Calif.:  RAND 
Corporation,  MG-670-AF,  2008. 

16  For  comprehensive  definitions  of  the  SAR  cost  categories,  see  DoD’s  Consolidated  Acquisition  Reporting  System 
(CARS)  Users  Guide ,  undated,  p.  126. 
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2.  schedule:  cost  variance  resulting  from  a  change  in  procurement  or  delivery 
schedule,  completion  date,  or  intermediate  milestone  for  development  or  pro¬ 
curement 

3.  support:  changes  in  program  cost  associated  with  training  and  training  equip¬ 
ment,  peculiar  support  equipment,  data,  operational  site  activation,  and  initial 
spares  and  repair  parts 

4.  economic:  cost  variance  resulting  from  price-level  changes  in  the  economy, 
including  changes  resulting  from  actual  escalation  that  differs  from  that  previ¬ 
ously  assumed  and  from  revisions  to  prior  assumptions  of  future  escalation 

5.  engineering:  cost  variance  resulting  from  an  alteration  in  the  physical  or  func¬ 
tional  characteristics  of  a  system  or  item  delivered  or  under  development  after 
establishment  of  such  characteristics 

6.  estimating:  cost  variance  resulting  from  correction  of  an  error  in  preparing  the 
baseline  cost  estimate,  refinement  of  a  prior  current  estimate,  or  a  change  in 
program  or  cost-estimating  assumptions  and  techniques 

7.  other:  changes  in  program  cost  resulting  from  natural  disasters,  work  stoppage, 
and  similarly  unforeseeable  events  not  covered  in  other  variance  categories. 

In  Figure  2.5,  we  show  the  cumulative  changes  for  the  SBIRS  program  in  the 
RDT&E  appropriation.  Notice  that  most  of  the  growth  is  attributed  to  the  “estimating 


Figure  2.5 
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category.”  But  does  this  mean  that  the  growth  was  solely  due  to  poor  estimating?  To 
check  this  hypothesis,  we  developed  our  own  set  of  variance  categories: 

•  bookkeeping:  changes  that  result  from  moving  funds  between  one  account  or 
another 

•  estimating:  changes  resulting  from  revisions  to  the  estimating  assumptions  or 
techniques 

•  external:  changes  resulting  from  external  reasons,  such  as  congressional  action  or 
money  moved  into  or  out  of  the  program  to  fund  other  priorities 

•  inflation:  changes  based  on  inflation/escalation  assumptions 

•  requirement:  changes  resulting  from  different  adding  or  removing  capability 
(additions  or  subtractions) 

•  schedule:  cost  changes  resulting  from  modification  to  the  schedule,  such  as  delays 
or  extending  production 

•  technical:  changes  to  cost  resulting  from  technical  reasons,  such  as  redesign 
needed  to  meet  original  technical  capability 

•  unknown:  changes  that  cannot  be  attributed  to  any  of  the  specific  reasons  given 
above. 

For  each  variance,  we  assigned  each  of  the  individual  explanations  to  a  specific 
category  using  our  best  interpretation  of  the  reason  stated.  In  Figure  2.6,  we  show  the 
same  cumulative  growth  for  the  SBIRS  RDT&E  using  this  different  categorization 

Figure  2.6 

Revised  Cost  Variance  Categories  Also  Uninformative 


0 

~o 


3 

E 

3 

U 


-1,000 


1997  1998  1999  2001  2002  2003  2004  2005  2006  2007  2009  2010 


SAR  year 


RAND  MG1 7 71/5-2.6 


72  DoD  and  Commercial  Advanced  Waveform  Developments  and  Programs  with  Nunn-McCurdy  Breaches 


of  the  individual  cost  variances.  Note  now  that  technical  and  schedule  explanations 
become  more  prominent  using  these  new  categories.  Very  few  issues  were  purely  esti¬ 
mating,  unlike  the  trend  shown  in  the  previous  figure.  More  troubling  is  that  in  a  large 
fraction  of  the  growth,  it  was  not  possible  to  attribute  the  growth  to  a  specific  category. 
In  the  end,  neither  of  these  cost  variance  approaches  is  very  illuminating.  So,  we  did 
not  pursue  this  avenue  further  in  the  analysis. 

SBIRS  High  entered  its  engineering  and  manufacturing  development  (EMD) 
phase  in  1996.  In  the  December  1997  SAR,  the  program  was  already  reporting  fund¬ 
ing  and  technical  problems  with  a  budget  shortfall  in  FY  1998  that  resulted  from  the 
need  to  accelerate  some  of  the  work  to  support  the  program’s  time  line  and  because 
of  congressionally  mandated  cuts.  The  program  office  chose  to  modify  the  contract  to 
address  part  of  the  shortfall.  This  action  delayed  increment  2,  all  five  GEO  satellites, 
and  HEO  sensor  delivery. 

From  1997  until  the  first  Nunn-McCurdy  breach  in  2001,  the  program  expe¬ 
rienced  a  series  of  APB  breaches.  The  following  is  the  explanation  provided  in  the 
December  1999  SAR  for  the  breaches: 

Schedule  and  Cost  breached  due  to  Air  Force  two  year  delay  to  SBIRS  High  and 
were  previously  reported  in  both  the  December  31,  1998,  and  the  September  30, 

1999,  SARs.  On  December  17,  1999,  The  SBIRS  Program  Office  issued  a  modifi¬ 
cation  that  reflected  the  contract  restructure.  At  the  same  time,  an  Undefinitized 
Contract  Action  option  was  issued  for  the  advanced  production  buy  for  GEO  3-5 
beginning  in  FY02.  The  SBIRS  High  data  now  incorporates  both  of  these  actions. 

As  a  result,  the  SBIRS  High  Acquisition  Program  Baseline  is  being  updated  to 
reflect  these  events.17 

The  December  2001  SAR  reported  the  program’s  first  Nunn-McCurdy  breach. 
SBIRS  High  breached  the  critical  threshold  of  the  PAUC  against  the  current  baseline. 

At  Secretary  of  the  Air  Force  (SECAF)  direction  and  in  concert  with  the  prime 
contractor,  Lockheed  Martin  Space  Systems  Company  (LMSSC),  an  Independent 
Review  Team  (IRT)  was  formed  to  review  the  program  and  diagnose  the  root 
causes  and  contributing  factors  of  the  significant  cost  growth.  Findings  from  the 
IRT  are  1)  the  SBIRS  program  was  too  immature  to  enter  System  Design  and 
Development;  2)  the  system  decomposition  and  flow  down  was  not  well  under¬ 
stood  as  the  program  continued  to  evolve;  and  3)  there  was  a  significant  break¬ 
down  in  execution  management.18 


17  “SBIRS  HIGH  Selected  Acquisition  Report  (SAR),”  Defense  Acquisition  Management  Information  Retrieval 
(DAMIR),  December  31,  1999,  p.  8. 

18  “SBIRS  HIGH  Selected  Acquisition  Report  (SAR),”  Defense  Acquisition  Management  Information  Retrieval 
(DAMIR),  December  31,  2001,  p.  26. 
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Using  the  results  of  the  analysis  of  the  breach,  the  program  office  decided  to 
change  how  the  program  was  being  managed.  Initially,  the  program  was  being  man¬ 
aged  using  a  Total  System  Performance  Responsibility  (TSPR)19  acquisition  strategy  at 
contract  award,  but  the  program  office  decided  that  the  program  was  too  complex  for 
this  strategy.  The  program  office  assumed  the  TSPR  responsibilities  after  the  clause  was 
taken  out  of  the  contract.  The  program  office  also  reasoned  that  government  oversight 
would  increase  after  removing  TSPR  from  the  contractor.  A  Program  Management 
Board  (PMB)  was  also  established  as  a  way  to  monitor  changes  to  the  program,  par¬ 
ticularly  changes  involving  budgeting,  technical,  and  schedule  assumptions.  Table  2.2 
provides  more  detail  on  the  challenges  and  actions  taken  to  mitigate  those  challenges 
for  the  first  Nunn-McCurdy  breach.  The  action  statements  are  taken  directly  or  para¬ 
phrased  from  the  SARs. 

As  the  program  was  being  restructured,  the  critical  breach  was  still  listed  in  the 
June  2002  SAR.  By  2003,  the  breach  was  resolved  and  the  program  proceeded  for¬ 
ward.  However,  in  the  quarterly  September  2003  SAR,  the  program  experienced  a 
schedule  APB  breach  as  the  HEO  Sensor  1  Delivery  and  HEO  Message  Certification 
were  behind  schedule.  This  required  another  revised  APB  for  the  program. 

In  2004,  the  program  reported  more  APB  breaches  and  a  significant  Nunn- 
McCurdy  breach  in  the  June  quarterly  SAR.  The  Nunn-McCurdy  breach  was  a  sig¬ 
nificant  PAUC  breach  against  the  current  baseline  in  addition  to  four  APB  breaches. 
The  following  is  the  explanation  or  the  breaches  provided  in  the  SAR: 

The  current  SAR  reflects  four  additional  schedule  breaches  resulting  from  the 
Program  Office  Estimate  replan:  Geosynchronous  Earth  Orbit  (GEO)  Satellite 
1  Delivery,  GEO  Satellite  2  Delivery,  GEO  Message  Certification  and  Mission 
Control  Station  (MCS)  Increment  2  Certification.  The  Program  Office  will  submit 
a  revised  APB  by  August  31,  2004.  The  revised  APB  will  be  based  on  the  June 
30,  2004,  SAR  cost  and  schedule  assumptions.  EiEO  Message  Certification:  This 
schedule  event  was  affected  by  the  late  delivery  of  the  EiEO  payloads  and  ground 
software  development.  GEO  Message  Certification/MCS  Increment  2  Certifica¬ 
tion:  These  schedule  events  were  affected  by  the  late  delivery  of  the  GEO  satellites 
and  ground  software  development.  APB  Cost  (RDT&E,  O&M,  and  PAUC)  and 
Nunn-McCurdy  Unit  Cost  (PAUC)  were  breached;  additional  GEO  1  and  2  hard¬ 
ware  delivery  and  Signal  Processing  Assembly  software  development  delays  have 
extended  the  procurement  time,  resulting  in  cost  growth  to  the  RDT&E,  O&M 
and  PAUC  of  the  SBIRS  program.20 


19  A  strategy  where  system  performance  responsibility  is  given  to  the  contractor  and  the  government  has  reduced 
oversight  responsibilities.  See,  for  example,  Henry  P.  Pandes,  A  Quest  for  Efficiencies:  Total  System  Performance 
Responsibility,  dissertation,  Air  Command  and  Staff  College,  Air  University,  April  2001. 

20  “SBIRS  HIGH  Selected  Acquisition  Report  (SAR),”  Defense  Acquisition  Management  Information  Retrieval 
(DAMIR),  June  30,  2004,  p.  6. 
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Table  2.2 

SBIRS  High  First  Breach  Actions 


Issues  Actions 


Technical  Risk  and  Maturity  Area 


SBIRS  program  was  too  immature  to  enter  Institute  independent  risk  assessment  of  programs  entering 
system  design  and  development  (SDD)  design  and  development. 

Scope  and  Requirements  Area 


The  system  decomposition  and  flow  down  Major  Command  should  be  responsible  for  the  detailed 
was  not  well  understood  as  the  program  description  of  the  expectations  associated  with  each  of  the 
continued  to  evolve  top-level  requirements  and  assist  the  developer  in  producing 

a  Technical  Requirements  Document. 

Work  towards  a  final  design  review  of  the  system  in  order  to 
close  out  the  liens  from  the  Critical  Design  Review  (CDR)  and 
to  ensure  the  maturity  of  the  program  to  proceed  further. 

Unstable  requirements  baseline  Established  a  flag-level  executive  committee  consisting  of 

acquisition  and  operational  expertise  from  the  government 
and  contractor,  which  has  oversight  of  execution  and  of 
requirements  flow  management.  The  executive  committee 
has  the  authority  to  adjudicate  cost,  schedule,  and 
performance  issues  associated  with  requirements  trades  and 
includes  all  mission  area  stakeholders. 

Program  restructured  to  an  evolutionary  block  modification 
strategy  that  phases  in  prioritized  requirements  in  a  well- 
defined  manner,  controlled  through  the  executive  committee 
process. 

Have  warfighters  assess  operational  risks  and  prioritize 
requirements. 

Content  baseline  has  been  put  under  program  office 
management  control. 


Development  and  Acquisition  Strategy  Area 


Software  development  strategy  too  Implementation  of  a  lower  risk  ground  software  approach 

complex  that  breaks  up  a  single  large  development  and  transition 

to  operations  into  multiple  block  deliveries  in  concert  with 
mission  needs  and  an  achievable  schedule  (spiral  approach). 

Contract  strategy  (TSPR)  inappropriate  for  Removed  the  TSPR  clause  from  the  contract.  The  program 
level  of  complexity  for  program  office  resumed  leadership  of  functions  that  had  been 

relinquished  to  the  contractor  under  TSPR. 
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Table  2.2  (Continued) 


Issues  Actions 


Performance  Management  Area 


Poor  management  performance  SBIRS  management  has  been  strengthened  (contractor  and 

government). 

System  engineering  resources  at  the  contractor,  as  well 
as  within  the  government  program  office,  have  been 
significantly  increased. 

Additional  cost  control  measures  include  augmenting  the 
Contract  Funds  Status  Report  with  a  detailed  report  of 
monthly  budget,  forecast  and  expenditure  per  product, 
Integrated  Product  Team,  and  total  program. 

The  SPO  is  implementing  an  improved  Earned  Value 
Management  (EVM)  process,  combining  traditional  EVM 
metrics  with  Defense  Contract  Management  Agency  (DCMA), 
contractor,  project  officer,  engineer  and  cost  analyst 
assessments. 

Control  of  a  disciplined  process  is  being  established.  This 
includes  periodic  independent  reviews,  annual  estimate  at 
completion  updates,  a  revised  award  fee  structure,  and  new, 
meaningful  metrics  that  measure  program  executability. 


Cost  Estimating  Area 


High  unit  cost  growth  Revised  estimating  methodology  (vague). 

Additional  cost  control  measures  include  augmenting  the 
Contract  Funds  Status  Report  with  a  detailed  report  of 
monthly  budget,  forecast  and  expenditure  per  product,  IPT, 
and  total  program. 


Schedule  Area 


Schedule  slip  The  newly  implemented  PMB  acts  as  the  decision  gate 

and  authority  to  approve  content  and  disposition  of  cost 
and  schedule  variances.  This  process  will  help  to  contain 
requirements  and  content  growth. 


SOURCES:  "SBIRS  HIGH  Selected  Acquisition  Reports"  and  Acquisition  Decision  Memoranda. 

The  2004  problems  were  not  fully  rectified.  In  2005,  the  program  experienced 
another  critical  Nunn-McCurdy  APUC  breach  against  the  current  baseline  and  a 
significant  Nunn-McCurdy  PAUC  breach  against  the  current  baseline.  The  PAUC 
increase  was  attributed  to  several  factors  associated  with  the  development  effort  and 
future  production  satellites  (GEO  3-5).  The  program  also  experienced  a  growth  in  the 
PAUC  as  a  result  of  manufacturing,  schedule,  and  risk  issues  with  the  EMD  contract. 
Schedule  extensions  as  the  result  of  manufacturing  problems  and  insufficient  forecasts 
were  also  contributors  to  the  breaches.  The  problem  took  the  following  additional  mea¬ 
sures  to  control  cost: 
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Each  month,  the  EVM  System  (EVMS)  information  is  reviewed  and  analyzed  by 
both  technical  and  EVM  analysts  in  the  program  office.  Results  are  briefed  to  the 
Segment  Program  Managers  and  SPD,  and  reported  in  the  Monthly  Acquisition 
Report  and  the  Defense  Acquisition  Executive  Summary.  The  contractor  team  also 
has  weekly  IPT/Segment  status  reviews  and  monthly  PMR  and  EVM  reviews.  The 
weekly  IPT/Segment  meetings  focus  on  current  issues,  schedule  and  cost  perfor¬ 
mance,  head  count  data,  and  business  risks  and  opportunities.  The  monthly  PMR 
addresses  EVM  data,  schedule  performance  and  technical  issues.  The  monthly 
EVM  meeting  addresses  the  earned  value  performance  for  the  month,  which 
includes  Schedule  Performance  Index  and  Cost  Performance  Index  data,  variance 
explanations  and  corrective  actions.21 

Table  2.3  provides  additional  details  on  the  challenges  faced  by  the  SBIRS  High 
program  after  the  second  set  of  Nunn-McCurdy  breaches  (actions  are  quotes  or  para¬ 
phrases  from  the  SARs). 

After  the  2005  Nunn-McCurdy  breach,  no  additional  Nunn-McCurdy  breaches 
have  occurred  for  the  program;  however,  the  program  had  APB  schedule,  RDT&E 
cost,  and  procurement  cost  breaches  in  both  2009  and  2010.  The  program  office  attrib¬ 
uted  the  schedule  breach  to  technical  issues  with  the  flight  software  system  (FSS)  and 
other  hardware  issues  that  caused  the  program  to  change  the  delivery  dates  on  the 
GEO  Satellite  1  and  2.  There  was  also  a  delay  in  the  Mission  Control  Station  (MCS) 
Increment  2  Certification.  APB  cost  breaches  are  attributed  to  the  addition  of  GEO 
Satellite  4  to  the  baseline  program  in  the  FY  2009  President’s  budget  (PB),  and  fund¬ 
ing  for  GEO  Satellite  5  and  6  was  also  added  to  the  program’s  budget  in  the  FY 
2011  PB.  “The  cost  deviation  against  RDT&E  appropriation  is  due  to  the  additional 
costs  required  to  complete  the  SBIRS  EMD  program  as  a  result  of  schedule  delays,  as 
well  as  the  additional  costs  required  to  implement  the  revised  SBIRS  ground  delivery 
strategy.”22 

Key  Takeaways  from  SBIRS  Review 

Two  factors  drove  the  repeated  breaches  in  the  SBIRS  High  program,  and  they  took 
years  to  fix: 

•  a  lack  of  technical  maturity  throughout  the  program  and  related  schedule  issues 

•  an  ineffective  acquisition  strategy  (TSPR)  that  led  to  significant  problems  in 
requirements  definition  and  stability. 


21  “SBIRS  HIGH  Selected  Acquisition  Report  (SAR),”  Defense  Acquisition  Management  Information  Retrieval 
(DAMIR),  September  30,  2005,  p.  27. 

22  “SBIRS  HIGH  Selected  Acquisition  Report  (SAR),”  Defense  Acquisition  Management  Information  Retrieval 
(DAMIR),  December  31,  2010,  p.  6. 
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Table  2.3 

SBIRS  High  Second  Breach  Actions 


Issues 

Actions 

Technical  Risk  and  Maturity  Area 

SBIRS  program  was  too 
immature  to  enter  SDD 

The  risk  to  the  "to  go"  program  has  been  reduced  by  adding  resources  to 
the  high  risk  elements  and  addressing  the  high  risk  elements  earlier  in  the 
development  phase. 

Addressing  the  key  technical  risks  by  adding  earlier  and  more  robust 
testing,  as  well  as  investigating  parallel  technology  paths. 

A  charter  is  in  draft  for  an  Independent  Review  Team  (IRT)  to  assess  the 
program  performance  three  times  a  year  and  report  to  the  Air  Force 
Program  Executive  Officer  for  Space  (AFPEO/Space)  on  progress  achieved 
and  future  risks. 

Independent  Program  Assessment  and  Independent  Costs  Assessment  to 
review  technical  and  cost  baselines. 

Revised  Test  and  Evaluation  Master  Plan  (TEMP)-testing  program 
inadequate. 

Scope  and  Requirements  Area 

Reduced  quantity  of  satellites. 

Development  and  Acquisition  Strategy  Area 

? 

Changed  from  block  to  annual  buys. 

Performance  Management  Area 

Poor  performance 

Several  significant  cost  control  enhancements  are  being  implemented  in  the 
program.  Among  these  are  the  transformation  of  the  contractor  Program 
Performance  Management  Process  (PPMP)  and  Program  Office  initiatives  to 
increase  government  insight  and  influence,  including  restructuring  the  joint 
government-contractor  surveillance  program. 

Increased  emphasis  on  business  operations,  predictive  metrics  and  trending. 

Program  office  scheduling  group  and  contractor  team  have  developed  the 
Unified  Program  Plan  (UPP)  to  document  and  capture  interdependencies, 
government  functions,  and  other  items  not  previously  included  in  the  SBIRS 
Integrated  Master  Schedule  (IMS). 

Program  Office  is  taking  aggressive  action  to  increase  technical  and 
managerial  skills  through  increased  training,  as  well  as  acquiring  additional 
senior  level  staff.  (Moving  People  to  contractor  site,  increased  oversight  of 
production,  etc.). 
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Table  2.3  (Continued) 


Issues 

Actions 

Cost  Estimating  Area 

Cost  growth 

Several  significant  cost  control  enhancements  are  being  implemented  in  the 
program.  Among  these  are  the  transformation  of  the  contractor  Program 
Performance  Management  Process  (PPMP)  and  Program  Office  initiatives  to 
increase  government  insight  and  influence,  including  restructuring  the  joint 
government-contractor  surveillance  program. 

Unspecified  revisions  to  the  estimate. 

Independent  Program  Assessment  and  Independent  Costs  Assessment  to 
review  technical  and  cost  baselines. 

The  program  office  is  continuing  to  work  with  the  Air  Force  Cost  Analysis 
Agency  to  develop  an  Independent  Cost  Model  for  SBIRS.  The  Engineering 
and  Manufacturing  Development  Cost  Model  was  validated  at  system  and 
subsystem  levels  in  December  2005.  Full  model  estimating  capabilities  will 
be  complete  in  January  2006. 

Scheduling  Area 

Schedule  slip 

System  Program  Director  established  an  organic,  independent  scheduling 
analysis  function  within  the  SPO.  Its  mission  is  to  analyze  the  contractor's 
program  schedule,  assess  credibility  and  completeness,  identify  missing 
or  improper  linkages,  and  provide  periodic  reports  to  the  senior  program 
office  staff. 

Externally  directed 
production  schedule 
delays  caused  an  increase 
in  parts  obsolescence  and 
extended  SPO  support. 

Second,  a  new  cost  estimate  was  developed  to  mitigate  the  parts 
obsolescence/parts  redesign  risk  associated  with  the  gap  between  GEO  2 
and  GEO  3. 

SOURCES:  "SB I RS  HIGH  Selected  Acquisition  Reports"  and  Acquisition  Decision  Memoranda. 


The  program  office  focused  on  controlling  the  problems  initially  through  such 
practices  as  better  earned  value  management  (EVM)  assessments.  However,  beyond  the 
technical  issues,  the  program  also  experienced  quantity  instability  that  was  imposed  on 
the  program  by  both  the  Air  Force  and  the  Office  of  the  Secretary  of  Defense  (OSD), 
leading  to  even  greater  instability.  Also  cited  were  multiple  budgeting  changes  from 
congressional  and  OSD  oversight. 

C-130  Avionics  Modernization  Program 

The  C-130  AMP  is  also  an  Air  Force  program.  The  system  “consolidates  and  installs 
the  mandated  Air  Force  Navigation/Safety  modifications,  the  Communications  Navi¬ 
gation  Surveillance/Air  Traffic  Management  (CNS/ATM)  capabilities,  and  the  C-130 
Broad  Area  Review  requirements  on  221  of  the  Air  Force’s  Combat  Delivery  C-130s. 
These  mandated  modifications  are  incorporated  with  various  other  reliability,  main¬ 
tainability,  and  sustainability  upgrades  to  include  installation  of  fleetwide  radars,  air¬ 
crew  displays,  dual  autopilots,  dual  flight  management  systems  and  HF/UHF/VHF 
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radios/data  links.  AMP  will  allow  this  fleet  complete  access  to  the  CNS/ATM-man- 
dated  national  and  international  air  space  for  the  foreseeable  future.”23  C-130  AMP 
was  awarded  its  Milestone  C  in  June  2010;  however,  the  FY  2013  PB  recommended 
its  cancelation. 

The  C-130  AMP  experienced  multiple  Nunn-McCurdy  breaches  because  it 
reported  two  unrelated  Nunn-McCurdy  breaches.  As  can  be  seen  in  Figure  2.7,  the 
first  was  a  significant  breach  in  2005  that  evolved  into  a  critical  Nunn-McCurdy 
breach  of  both  the  APUC  and  PAUC  against  both  the  current  and  original  baselines 
in  2006.  This  breach  was  unrelated  to  the  change  in  Nunn-McCurdy  legislation  in 
2006  because  the  program  was  undergoing  several  issues  outside  the  legislative  change 
in  2006.  The  second  was  another  significant  Nunn-McCurdy  breach  of  the  APUC 
against  the  current  baseline  in  2009.  The  program  also  had  four  revised  APBs  after 
Milestone  B  in  2001. 24 

Figure  2.8  tracks  the  program’s  percentage  change  in  the  PAUC  and  APUC 
against  original  and  current  baselines  since  Milestone  B  was  awarded  in  2001  through 
December  2010  using  data  provided  in  various  annual  and  quarterly  SARs.  As  can  be 
seen  in  the  chart,  all  four  measures  of  unit  cost  grew  significantly  in  2006: 

•  The  PAUC  against  the  current  baseline  increased  116  percent, 

•  The  APUC  against  the  current  baseline  increased  98  percent. 

•  The  PAUC  against  the  original  baseline  increased  169.  and 

•  The  APUC  against  the  original  baseline  increased  120  percent. 

Figure  2.7 

C-130  AMP  Nunn-McCurdy  Breach  History 


Current  baseline 
Original  baseline 


1998  1999  2000  2001  2002  2003  2004  2005 


PAUC 
APUC 
PAUC  Hi 
APUC  : 


2007  2008  2009  2010 


New  APB  or  rebaseline 
Critical  breach 
Significant  breach 
Not  tracked 


SOURCE:  "C-130  AMP  Selected  Acquisition  Reports"  (1998  through  2010). 

RAND  MG1171I5-2.7 


23  “q_j3q  AMP  Selected  Acquisition  Report  (SAR),”  Defense  Acquisition  Management  Information  Retrieval 
(DAMIR),  December  31,  2010,  p.  3. 

24  The  Under  Secretary  of  Defense  (Acquisition,  Technology,  and  Logistics)  USD  (AT&L)  approved  the  pro¬ 
gram’s  Milestone  B  for  entry  into  SDD  on  July  27,  2001,  according  to  the  December  2001  SAR. 
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Figure  2.8 

C-130  AMP  PAUC  and  APUC  Trends 


SOURCE:  "C-130  AMP  Selected  Acquisition  Reports"  (2001  through  2010). 

RAND  MG1171/5-3.8 


After  Milestone  B  in  2001,  the  ACAT  1C  program  immediately  experienced 
some  funding  issues,  technical  difficulty,  and  cost  growth  in  the  RDT&E  portion  of 
the  program.  All  these  issues  persisted  throughout  the  development  phase. 

In  2001,  the  program  office  reported  that  there  was  not  enough  funding  to  meet 
the  training  system  requirements  in  the  C-130  AMP  ORD.  The  program  also  reported 
two  APB  breaches  in  the  December  2001  SAR:  an  RDT&E  breach  and  a  PAUC  APB 
breach  signaling  some  difficulties  with  cost  control.  The  program  also  recognized  that 
it  would  not  be  able  to  meet  the  2005  Global  Air  Traffic  Management  requirement.  A 
reason  given  for  the  delay  was  that  only  a  limited  number  of  aircraft  could  be  modified 
simultaneously. 

The  program  provided  some  explanation  for  the  growing  cost  and  schedule  prob¬ 
lems  in  the  December  2002  SAR: 

As  a  result  of  funding  reductions  in  FY03  and  FY04,  C-130  AMP  has  had  to  slow 
down  the  ramp  up  of  developmental  activities.  This  slow  down  has  resulted  in  a 
change  to  APB  milestone  dates  and  an  increase  in  the  cost  of  the  program.  An 
Engineering  Change  Proposal  (ECP)  replan  is  currently  being  evaluated  and  a 
revised  APB  is  being  staffed  to  reflect  the  proposed  milestone  dates.  ...  In  addi¬ 
tion  to  the  schedule  breach  we  have  an  11  percent  cost  increase  in  the  RDT&E 
appropriation  due  to  the  additional  cost  to  AMP  associated  with  the  stretchout  of 
the  program  and  the  acceleration  of  the  Common  Avionics  Architecture  for  Pen- 
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etration  (CAAP)  portion  of  the  C-130  AMP  program.  The  cost  increases  to  the 
RDT&E  line  result  in  an  APB  cost  breach  to  RDT&E  and  Program  Acquisition 
Unit  Cost.  A  Program  Deviation  Report  (PDR)  is  being  staffed  and  coordinated 
for  both  the  schedule  and  cost  breaches.25 

The  program  did  not  report  any  breaches  in  2003;  however,  the  program  office 
once  again  warned  about  growing  cost  problems  in  the  out-years  that  were  discussed  in 
an  independent  cost  estimate  (ICE).  The  ICE  pointed  out  that  the  introduction  of  new 
requirements,  changes  in  the  cost  estimating  methodology,  and  projected  increased 
depot  labor  rates  for  AMP  kit  installation  mainly  drove  the  cost  growth.  The  result  of 
cost  increases  in  2001,  2002,  and  2003  was  three  new  APBs,  which  were  presented  in 
each  of  these  years. 

The  C-130  AMP  program’s  cost  growth  after  Milestone  B  breached  the  Nunn- 
McCurdy  significant  PAUC  unit  cost  threshold  against  the  original  baseline  when  the 
Nunn-McCurdy  legislation  was  amended  to  include  reporting  both  the  PAUC  and 
APUC  against  the  original  baseline.  The  program  reported  a  quantity  reduction  from 
519  to  434  and  a  loss  of  funding.26  The  quantity  reduction  was  due  to  the  Air  Mobility 
Command’s  decision  to  retire  C-130E  aircraft.  The  problem  also  had  a  mix  of  other 
issues  that  created  instability  as  indicated  below: 

The  ongoing  issues  stemming  from  funding  challenges,  protests  and  contractor 
overruns  all  factor  into  the  current  estimate  activity  and  efforts  supporting  the 
program  replan.  Protest  decisions  have  changed  acquisition  strategy  regarding  full- 
rate  production  competition  and  preliminary  estimates  reflect  these  costs  as  well. 
Complexities  encountered  during  initial  trial  modifications,  and  supplier  delivery 
delays  have  also  contributed  to  contract  cost  overruns.27 

In  2006,  the  program  faced  further  instability  as  the  program  breached  all  four 
Nunn-McCurdy  unit  cost  thresholds  at  the  critical  level.  The  breaches  can  be  attrib¬ 
uted  in  part  to  another  decrease  in  quantity  from  434  to  268  as  a  result  of  a  loss  of  U.S. 
Special  Operations  Command  (USSOCOM)  funding,  restructure  actions  to  realign 
the  program  budget,  and  increased  production  costs. 

In  June  2007,  the  USD  (AT&L)  certified  the  C-130  AMP  after  its  critical  Nunn- 
McCurdy  breach.  Some  of  the  stipulations  of  the  certification  included  those  listed 
below: 

•  The  program  was  transferred  from  ACAT  1C  to  ID  status  making  the  USD 
(AT&L)  the  Acquisition  Executive  over  the  program. 


25  “C-130  AMP  Selected  Acquisition  Report  (SAR),”  December  2002,  p.  5. 

26  “C-130  AMP  Selected  Acquisition  Report  (SAR),”  December  2005,  p.  6. 

27  “C-130  AMP  Selected  Acquisition  Report  (SAR),”  December  2005,  p.  24. 
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•  The  quantity  of  aircraft  was  reduced  to  222,  which  included  C-130  H2,  H2.5 
and  H3  Mission  Design  Series  aircraft. 

•  The  program  was  supposed  to  be  funded  to  the  OSD  Cost  Analysis  Improvement 
Group  (CAIG)  estimate  with  reprogramming  from  procurement  to  RDT&E. 

The  certification  also  called  for  reworking  the  acquisition  strategy  and  APB  and 
for  review  of  EVM.  In  2007,  the  program  worked  toward  Milestone  C  and  low-rate 
initial  production  (LRIP)  using  an  “aggressive”  schedule  that  was  outlined  by  the  USD 
(AT&L). 

Even  with  the  rebaselining  of  the  program  only  a  couple  of  years  before  2009, 
the  program  once  again  breached  a  Nunn-McCurdy  threshold  with  APB  breaches 
in  schedule,  procurement,  and  APUC.  The  Nunn-McCurdy  breach  was  a  significant 
APUC  breach  against  the  current  baseline.  The  cause  of  these  breaches  was  provided 
in  the  December  2009  SAR: 

For  Schedule  breach,  Milestone  C  Defense  Acquisition  Board  (DAB)  approval 
slipped  from  June  2008  to  March  2010,  causing  a  breach  to  the  APB.  Further 
refinement  of  production  acquisition  strategy  and  program  costs  was  required 
prior  to  granting  full  Milestone  C  DAB  approval.  The  current  estimate  for  Mile¬ 
stone  C  DAB  is  March  2010.  An  updated  Acquisition  Program  Baseline  (APB) 
will  follow.  For  Procurement  and  Average  Procurement  Unit  Cost  (APUC),  the 
Cost  Assessment  and  Program  Evaluation  (CAPE)  estimate  significantly  exceeds 
the  APB  threshold.  Contributors  to  the  increase  include  Spares  and  Training  Sys¬ 
tems  estimates,  Milestone  C  schedule  delays,  and  loss  of  FY  2010  procurement 
funding  due  to  the  Milestone  C  slip.28 

After  the  December  2009  breach,  The  Air  Force  wanted  to  cancel  the  program; 
however,  Congress  wanted  it  to  continue.  After  a  couple  of  years  of  debate,  the  pro¬ 
gram  was  recommended  for  cancelation  in  the  FY  2013  budget  request. 

Table  2.4  provides  the  specific  challenges  and  actions  taken  to  mitigate  the  chal¬ 
lenges  since  Milestone  B.  The  shaded  rows  represent  significant  actions  taken  at  the 
second  breach. 

Key  Takeaways  from  C-130  AMP  Review 

The  C-130  AMP  program  was  beset  by  both  significant  quantity  cuts  as  the  program 
lost  support  and  congressional  cuts  to  funding  throughout  its  RDT&E  phase.  In  addi¬ 
tion,  a  contract  protest  was  the  result  of  acquisition  irregularities  and  led  to  a  major 
acquisition  strategy  change — from  sole  source  to  competitive  production  late  in  devel¬ 
opment.  Finally,  technical  issues  surfaced  late  into  design  in  which  the  retrofits  were 
more  complicated  than  anticipated  and  led  to  cost  and  schedule  delays. 


28 


‘C-130  AMP  Selected  Acquisition  Report  (SAR),”  December  2009,  p.  5. 
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Table  2.4 

C-130  AMP  SAR  Issues 


Issues 

Actions 

Technical  Risk  and  Maturity  Area 

Technical  complexity 
and  variation  of 
individual  aircraft 
higher  than  thought 

Redesign  wire  harness 

Immature  technical 
baseline 

Program  restructuring:  review  and  restructuring  delayed  LRIP  decision; 

Milestone  C  Defense  Acquisition  Board  (DAB)  approval  slipped  from  June  2008 
to  March  2010 

Scope  and  Requirements  Area 

Unstable  partnerships 

USSOCOM  and  Sweden  buy  into  program;  rescission  by  USSOCOM  in  FY  2007 
(partial  in  FY  2006)  and  Sweden  in  FY  2010 

Acceleration  of  Common  Avionics  Architecture  Program  (CAAP)  for  USSOCOM 

Development  and  Acquisition  Strategy  Area 

GAO  review  after 
protests  as  a  result  of 
a  criminal 
investigation 

Change  to  strategy  from  sole  source  to  competed  installation  kits — meant 
extending  schedule  for  contractor  acculturation  (change  full  rate  production 
(FRP)  to  open  competition;  added  source  familiarization  plan).  GAO  imposed  this 
change. 

Performance  Management  Area 

Initially  funded  as 
an  ACAT  1C 

Changed  to  ACAT  ID  after  initial  breach. 

Cost  Estimating  Area 

Unstable  costs 

Develop  service  cost  position  (started  in  FY  2005  but  not  really  complete  until 

FY  2006);  later,  OSD  pushed  to  use  CAIG  estimate  after  first  breach  (FY  2008);  at 
second  breach,  program  was  funded  to  CAPE  estimate. 

A  program  replanning  was  in  the  works  that  was  designed  to  reconcile 
remaining  development  with  funding.  Rebaselined  in  FY  2006. 

Boeing  did  not  commit  to  out-year  pricing. 

Revised  cost 
methodologies 

December  2003:  The  C-130  AMP  began  an  ICE;  December  2004:  The  C-130 

AMP  (as  of  the  December  2010  SAR)  is  currently  developing  a  service  cost 
position  (SCP)  in  conjunction  with  the  Air  Force  Cost  Analysis  Agency  to 
update  development  and  production  cost  estimates.  Preliminary  SCP  findings 
are  projecting  increases  to  program  cost  in  FY  2007  and  beyond  as  a  result  of 
forecasted  increases  in  production  rates,  depot  installation  costs,  and  potential 
risks  in  development  test. 

Funding  Area 

Unstable  funding 

Funding  reductions  in  FY  2003  and  FY  2004  (December  2002  SAR);  SDD,  LRIP, 
and  production  readiness  review  (PRR)  were  delayed  two  years  as  a  result. 

Congressional  actions 

Zeroed  funding  for  FY  2010  in  FY  2009. 

Training  line  under-  Transfer  of  training  funds  to  RDT&E  (FY  2005)  "Zero  Based  Transfer" 
funded  at  Milestone  B 
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Table  2.4  (Continued) 


Issues 

Actions 

Schedule  Area 

Availability  of  aircraft 

Stretch  out  of  production  schedule. 

SOURCES:  "C-130  AMP  Selected  Acquisition  Reports"  and  Acquisition  Decision  Memoranda. 


H-1  Upgrades  (4BW/4BN) 

The  H-1  Upgrades  program  is  a  U.S.  Marine  Corps  (Navy)  midlife  upgrade  to  the 
AH-1W  attack  helicopter  and  the  UH-1N  utility  helicopter. 

The  mission  of  the  AH-1Z  attack  helicopter  is  to  provide  rotary  wing  close  air 
support,  anti-armor,  armed  escort,  armed/visual  reconnaissance  and  fire  support 
coordination  capabilities  under  day/night  and  adverse  weather  conditions  for  the 
United  States  Marine  Corps  (USMC).  The  mission  of  the  UH-1Y  utility  helicopter 
is  to  provide  command,  control  and  assault  support  under  day/night  and  adverse 
weather  conditions.  Both  the  AH-1Z  and  UH-1Y  aircraft  incorporate  state  of  the 
art  designs,  which  serve  to  improve  capability,  lethality  and  survivability.  Major 
modifications  include  a  new  four-bladed  rotor  system  with  semi-automatic  blade 
fold  of  the  new  composite  rotor  blades,  new  performance  matched  transmissions, 
a  new  four-bladed  tail  rotor  and  drive  system,  upgraded  landing  gear,  and  pylon 
structural  modifications.  The  H-1  Upgrades  aircraft  have  increased  maneuver¬ 
ability,  speed,  and  payload  capability.  Both  aircraft  have  fully  integrated  common 
cockpits/avionics  that  reduce  operator  workload  and  improve  situational  aware¬ 
ness,  thus  increasing  safety.29 

The  program  was  awarded  its  Milestone  C  in  September  2008,  12  years  after  it 
was  awarded  its  Milestone  II  in  1996. 

The  H-1  Upgrades  program  is  also  a  repeat  Nunn-McCurdy  breach  program  (see 
Figure  2.9).  It  first  experienced  a  critical  Nunn-McCurdy  breach  in  2001.  At  that  time, 
the  program  had  a  breach  against  the  current  baseline  for  both  the  PAUC  and  APUC. 
However,  it  did  not  breach  again  until  2008,  when  the  program  experienced  a  signifi¬ 
cant  PAUC/APUC  unit  cost  growth  breach  against  the  current  baseline.  In  addition  to 
the  breaches,  the  program  has  revised  its  APB  four  times  from  1998  through  2010.  It 
is  interesting  to  note  that  the  program  has  never  experienced  a  Nunn-McCurdy  breach 
against  its  original/revised  original  baseline. 

Figure  2.10  tracks  the  program’s  percentage  change  in  the  PAUC  and  APUC 
since  Milestone  B  was  awarded  in  1996  using  data  provided  in  various  annual  and 
quarterly  SARs.  As  can  be  seen  in  the  figure,  the  program  experienced  its  largest  cost 
growth  in  2001: 


29  "H-1  UPGRADES  (4BW/4BN)  Selected  Acquisition  Report  (SAR),”  Defense  Acquisition  Management 
Information  Retrieval  (DAMIR),  December  31,  2010,  p.  3. 
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Figure  2.9 

HI  Upgrades  Nunn-McCurdy  Breach  History 


Current  baseline 
Original  baseline 


1998  1999  2000  2001  2002  2003  2004 


PAUC 
APUC 
PAUC  gH 
APUC 


2006  2007  2008  2009  2010 


New  APB  or  rebaseline 
Critical  breach 
Significant  breach 
Not  tracked 


SOURCE:  "HI  Upgrades  Selected  Acquisition  Reports"  (1998  through  2010). 

RAND  MG1 17115-2.9 


Figure  2.10 

H-1  Upgrades  PAUC  and  APUC  Trends 


1997  1998  1999  2001  2001  2002  2003  2004  2004  2005  2005  2006  2007  2008  2008  2009  2010 


SOURCE:  "SHI  Upgrades  Selected  Acquisition  Reports"  (1998  through  2010). 

RAND  MG1171/5-2.10 


•  The  PAUC  against  the  current  baseline  increased  53  percent, 

•  The  APUC  against  the  current  baseline  increased  50  percent. 

The  program  has  also  had  some  volatility  in  its  unit  cost  growth  metrics  over 
time.  There  are  other  notable  peaks: 


86  DoD  and  Commercial  Advanced  Waveform  Developments  and  Programs  with  Nunn-McCurdy  Breaches 


•  In  1999,  the  PAUC  grew  14  percent  and  the  APUC  grew  nearly  13  percent  against 
the  current  baseline. 

•  In  2004,  the  PAUC  grew  12  percent  and  the  APUC  grew  nearly  11  percent 
against  the  current  baseline. 

•  In  2010,  the  PAUC  grew  22  percent  and  the  APUC  grew  nearly  27  percent  against 
the  revised  original  baseline. 

The  HI  Upgrades  program  was  approved  entry  into  EMD  in  1996.  As  reported 
in  SARs  from  1997  through  2001,  the  program  experienced  a  series  of  APB  breaches 
signaling  multiple  problems  that  the  program  was  experiencing  during  its  early  EMD 
phase  primarily  in  regard  to  schedule  and  cost: 

•  December  1997  SAR:  schedule  and  performance  breaches 

•  December  1998  SAR:  schedule  breach 

•  December  1999  SAR:  schedule,  RDTE,  procurement,  PAUC,  and  APUC 
breaches 

•  September  2001  SAR:  schedule  and  RDTE  breaches. 

The  program  reported  a  $58.8  million  EMD  program  shortfall  in  1999,  which 
required  that  the  program  office  pursue  funding  sources  for  the  estimate  at  completion 
(EAC)  shortfall  in  FY  2000  and  FY  2001.  The  program  used  above  threshold  repro¬ 
gramming  (ATR)  and  the  Navy’s  acquisition  stability  reserve  (ASR)  fund  to  mitigate 
the  shortfall  along  with  cost-reduction  initiatives.  The  program  also  reported  the  APB 
breaches  for  RDT&E  and  procurement  funding.  The  program  office  said  that  the  rea¬ 
sons  for  these  breaches  were  based  on  the  following: 

1)  Previously  approved  program  changes  such  as  the  UH-1Y  common  cockpit; 
crashworthy  AH-1Z  crew  seats;  ground  proximity  warning  system  (GPWS);  and 
Integrated  Mechanical  Diagnostic  development  and  production.  2)  Estimate  at 
Completion  (EAC)  growth  caused  by  increased  contractor  rates,  less  reused  air¬ 
craft  structure  (than  estimated),  and  underfunded  logistics  elements.  A  revised 
Acquisition  Program  Baseline  is  in  process  to  support  the  new  Acquisition  Strategy 
and  cost  increases.30 

The  schedule  delays  worsened  after  1999,  which  forced  the  program  to  report 
delays  of  more  than  six  months  in  a  September  2001  quarterly  SAR.31  The  program 
office  decided  on  a  program  restructure  that  would  address  both  schedule  and  cost 
overruns.  The  new  baseline  also  added  “additional  time  for  identified  technical,  test, 
and  logistics  risk  mitigation  plans  prior  to  Operational  Evaluation  (OPEVAL),  and 


30  “H-l  UPGRADES  (4BW/4BN)  Selected  Acquisition  Report  (SAR),”  December  30,  2001,  p.  5. 

31  This  problem  was  also  likely  in  2000,  but  no  SARs  were  published  in  2000. 
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adds  additional  government  support  to  recover  decrements  in  prior  fiscal  years.”32  At 
the  time,  the  program  office  warned  that  the  program  restructure  would  result  in  a  sig¬ 
nificant  increase  in  both  the  PAUC  and  APUC,  shown  by  detailed  reviews  of  costs  by 
the  contractor  and  Naval  Air  Systems  Command  (NAVAIR).  This  report  in  September 
2001  came  only  a  couple  of  months  before  the  program  reported  its  critical  Nunn- 
McCurdy  breach  of  both  the  PAUC  and  APUC  against  the  current  baseline. 

The  program  office  submitted  its  Program  Deviation  Report  (PDR)  in  October 
2001  as  a  result  of  the  above  RDT&E  cost  and  schedule  breaches.  At  the  same  time, 
Bell  Helicopter  Textron  (BHTI)  conducted  an  internal  review  because  there  was  con¬ 
cern  about  deteriorating  contractor  performance.  NAVAIR  and  the  CAIG  were  also 
tasked  to  examine  contractor  performance  and  to  provide  a  revised  EAC.  After  these 
reviews,  the  H-l  Upgrades  program  proposed  a  program  restructure  described  above. 

Twenty  percent  of  both  the  critical  PAUC  and  APUC  breach  was  attributed 
to  EMD  cost  growth.  The  other  80  percent  was  the  result  of  a  production  estimate 
update.  The  program  office  provided  a  breakdown  in  the  December  2001  SAR  of  the 
more  specific  reasons  for  the  breach  as  indicated  in  Table  2.5. 

In  an  effort  to  control  costs,  the  program  office  focused  on  revising  the  Engineer¬ 
ing  Mockup  Unit  (EMU),  which  was  causing  engineering  drawings  to  be  late  and 
therefore  EMD  costs  to  rise.  The  program  office  also  focused  on  improving  forecast¬ 
ing  and  predictive  change  tools  by  improving  contractor  EVM  tools,  processes,  proce¬ 
dures,  and  training.  To  limit  production  cost  growth,  the  program  office  reviewed  the 
production  estimate  in  detail  to  get  a  more  accurate  estimate  for  budgeting.  Specifi¬ 
cally,  the  program  office  took  the  following  measure  to  eliminate  future  schedule  and 
cost  problems: 

The  contractor  and  government  have  staffed  a  production  team  to  analyze  and 
implement  transition  to  production  cost  control  and  efficiency  measures.  The  con¬ 
tractor  has  invested  in  implementing  Advanced  Planning  and  Scheduling  (APS) 
and  Component  and  Supplier  Management  (CSM)  programs  to  manage  and  con¬ 
trol  costs.  The  business  case  analysis  for  these  programs  includes  commitments 
for  direct  material  cost  savings,  inventory  reduction,  administrative  and  equip¬ 
ment  utilization  efficiencies  as  well  as  improvements  in  manufacturing  and  design 
productivity.  The  revised  EMD  acquisition  strategy  includes  increased  contract 
cost  control  measures.  The  government  is  currently  negotiating  an  EMD  con¬ 
tract  modification  that  includes  a  production  price  commitment  curve  for  the  first 
two  LRIP  lots.  In  addition,  the  revised  acquisition  strategy  includes  an  EMD 
performance  based  incentive  structure  if  the  contractor  achieves  predetermined 
scheduled  or  performance  milestone  events  and  EVM  performance  goals  to  fur¬ 
ther  control  costs.33 


32  “H-l  UPGRADES  (4BW/4BN)  Selected  Acquisition  Report  (SAR),”  December  2001,  p.  4. 

33  “H-l  UPGRADES  (4BW/4BN)  Selected  Acquisition  Report  (SAR),”  December  31,  2001,  p.  21. 
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Table  2.5 

Reasons  for  the  H-1  Upgrades  2001  Critical  PAUC/APUC  Breach  Against  the  Current  Baseline 


PAUC  APUC 


1.  Material  Update  Based  on  EMD  Actuals — 21%: 
Original  material  estimates  were  developed  from 
a  parts  list  provided  by  the  contractor  based  on 
similar  programs.  Three  of  the  five  EMD  aircraft 
have  now  completed  manufacturing  and  are  in 
flight  test  (as  of  the  2010  SAR).  The  current 
estimate  is  based  upon  this  information  and 
comparisons  with  analogous  systems. 

2.  Learning  Curve  Update  Based  on  V-22/lndustry 
Trend — 20%:  Previous  labor  and  material  learning 
curve  projections  were  based  on  AH-1W  data.  The 
learning  curves  have  been  updated  based  on  V-22 
actuals  and  updated  industry  trends  that  are  much 
flatter  than  the  previous  projections. 


1.  Material  Update  Based  on  EMD  Actuals — 27%: 
Original  material  estimates  were  developed  from 
a  parts  list  provided  by  the  contractor  based  on 
similar  programs.  Three  of  the  five  EMD  aircraft 
have  now  completed  manufacturing  and  are  in 
flight  test.  The  current  estimate  is  based  upon 
this  information  and  comparisons  with  analogous 
systems. 

2.  Learning  Curve  Update  Based  on  V-22/lndustry 
Trend — 25%:  Previous  labor  and  material  learning 
curve  projections  were  based  on  AH-1W  data.  The 
learning  curves  have  been  updated  based  on  V-22 
actuals  and  updated  industry  trends  that  are  much 
flatter  than  the  previous  projections. 


3.  Increased  Support  Funds — 13%:  Shortly  after  the  3.  Increased  Support  Funds — 17%:  Shortly  after 
June  2000  APB  was  signed  significant  funding  was  the  June  2000  APB  was  signed,  significant  funding 
added  in  the  OPNAV  spares  requirement  generation  was  added  in  the  OPNAV  spares  requirement 
process  to  adequately  spare  to  an  85%  readiness  generation  process  to  adequately  spare  to  an 
goal.  In  addition,  FI-1  simulators  were  moved  into  85%  readiness  goal.  In  addition,  FI-1  simulators 
the  APN-1  program  from  APN-7  account.  Finally,  were  moved  into  the  APN-1  program  from  APN-7 
the  USMC  conducted  a  review  of  their  Simulator  account.  Finally,  the  U.S.  Marine  Corps  conducted 
Master  Plan  and  subsequently  doubled  the  number  a  review  of  their  Simulator  Master  Plan  and 
of  simulators  from  seven  to  14. 


4.  Contractor  Rate  Increases — 12%:  The  revised 
production  estimate  incorporates  the  current 
Forward  Pricing  Rate  Agreement  (FPRA)  dated 
December  2001.  The  updated  projections  are 
based  on  lower  forecasts  for  both  commercial  and 
military  business  including  reduced  V-22  and  FI-1 
buys. 

5.  Prime  Contractor  Performance — 10%:  The  FI-1 
upgrades'  contractor  significantly  underestimated 
the  design  and  development  tasks  primarily  in 
airframe  integration  and  software. 

SOURCE:  "H-1  UPGRADES  (4BW/4BN)  Selected  Acquisition  Report  (SAR),"  December  2001,  pp.  19-20. 


subsequently  doubled  the  number  of  simulators 
from  seven  to  14. 

4.  Contractor  Rate  Increases — 15%:  The  revised 
production  estimate  incorporates  the  current  FPRA 
dated  December  2001.  The  updated  projections 
are  based  on  lower  forecasts  for  both  commercial 
and  military  business  including  reduced  V-22  and 
H-1  buys. 


After  the  program  was  certified  to  continue  in  accordance  with  Nunn-McCurdy 
law,  it  did  not  have  another  Nunn-McCurdy  breach  until  2008;  however,  the  program 
continued  to  experience  APB  breaches  from  year  to  year: 

•  December  2004  SAR:  APB  schedule,  RDTE,  procurement,  PAUC,  APUC 
breaches 

•  December  2005  SAR:  APB  schedule  breach 

•  December  2006  SAR:  APB  schedule  breach 

•  December  2007  SAR:  APB  performance  breach. 
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In  June  2004,  the  program  reported  technical  problems  that  caused  a  sched¬ 
ule  delay  of  more  than  six  months.  According  to  the  December  2004  SAR,  “the  air¬ 
craft  experienced  higher  than  expected  structural  strength  degradation  due  to  engine 
exhaust  gas  impinging  on  the  tailboom  and  weakening  the  metal.  In  order  to  ensure 
safety  margins  were  maintained,  flight  testing  was  temporarily  suspended  February  5, 
2004.”34  The  program  office  modified  four  EMD  aircraft  as  a  temporary  solution  and 
then  produced  a  long-term  solution,  although  this  longer-term  solution  required  more 
funding.  Additional  time  was  added  to  the  schedule  because  developmental  testing 
identified  other  issues  that  included  technical  issues  with  rocket  gas  ingestion  and 
weapons  system  integration.  The  sum  of  these  issues  created  an  RDT&E  shortfall.  The 
program  reduced  H-l  procurement  quantities  by  five  and  converted  the  associated  air¬ 
craft  procurement  funding  to  RDT&E  as  a  solution. 

Even  with  the  above  solutions,  the  program  continued  to  have  an  APB  schedule 
breach  in  2005  because  of  the  OPEVAL  completion  date  that  was  not  met  and  the 
AH-1Z  Cruise  Speed  key  performance  parameter  (KPP)  that  also  was  not  met.  The 
cruise  speed  KPP  was  modified  as  a  result  of  this  issue  as  a  plan  to  move  forward.  These 
changes  and  other  issues  required  another  revised  acquisition  strategy  and  a  revised 
APB  in  2006. 

In  June  2008,  the  program  experienced  another  Nunn-McCurdy  breach.  This 
second  breach  included  a  significant  PAUC/APUC  breach  against  the  current  base¬ 
line.  The  breach  took  place  shortly  before  the  program  was  approved  for  Milestone  C 
in  September  2008.  The  following  is  the  program  office’s  explanation  for  the  Nunn- 
McCurdy  breach: 

The  H-l  Upgrades  program  has  deviated  from  the  approved  Acquisition  Pro¬ 
gram  Baseline  (APB).  The  Operational  Evaluation  (OPEVAL)  Phase  II  Complete 
(AH-1Z)  threshold  date  of  September  2008  will  not  be  met  due  to  unresolved 
Critical  Operational  Issues  (COIs)  related  to  AH-1Z  weapons  employment.  A 
Program  Deviation  Report  has  been  submitted.  A  revised  APB  is  being  prepared 
for  USD  (AT&L)  approval  and  will  be  presented  at  the  September  2008  Defense 
Acquisition  Board  (DAB).  As  previously  reported,  main  rotor  cuff  static  strength 
limitations  will  preclude  the  UH-1Y  from  satisfying  the  Maneuverability  Key  Per¬ 
formance  Parameter  (KPP)  requirement  range  of  -0.5  to  +2.5  (G’s).35 

The  program  office  was  able  to  satisfactorily  show  that  the  program  was  able  to 
deal  with  the  above  issues,  so  the  USD  (AT&L)  moved  forward  with  Milestone  C 
approval.  Table  2.6  provides  the  issues  that  the  program  faced  and  the  actions  it  took 
to  continue  with  the  program  after  constant  schedule,  contractor  performance,  and 


34  “H-l  UPGRADES  (4BW/4BN)  Selected  Acquisition  Report  (SAR),”  June  30,  2004,  p.  4. 

33  “H-l  UPGRADES  (4BW/4BN)  Selected  Acquisition  Report  (SAR),”  Defense  Acquisition  Management 
Information  Retrieval  (DAMIR),  June  30,  2008,  p.  5. 
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Table  2.6 

H-1  Upgrades  Issues 


Issues 

Actions 

Technical  Risk  and  Maturity  Area 

Late  drawings  to  manufacturing 

The  EMU  was  not  revised  to  adequately  model  the  upgrade 
systems.  The  restructure  provides  funds  necessary  to  completely 
revise  the  EMU  to  fully  model  the  AH-1Z  and  UH-1Y  upgrade 
systems,  to  ensure  that  the  engineering  analysis,  manufacture 
effects  and  efficiencies  are  realized  for  any  future  changes. 

Less  reused  aircraft  structure 
(than  estimated) 

Modified  design. 

Scope  and  Requirements  Area 

Unstable  requirements 

Revisions  to  cost  estimate. 

Increased  readiness  goal 

Added  additional  costs  for  spares  to  achieve  85%  readiness  goal. 

Development  and  Acquisition  Strategy  Area 


Combat  losses  of  H-1's. 

MSI  1 1  decision  to  build  new  aircraft  rather  than  remanufacture. 

Performance  Management  Area 

Poor  contractor  performance 

Initiated  OSD  CAIG  and  NAVAIR  independent  program  review. 

Overhauled  and  updated  EVM  though  use  of  outside  consultant. 

Renegotiated  contract  terms  to  incentivize  control  cost  and 
schedule. 

Designated  as  a  "buy-to-budget"  program. 

NAVAIR  program  office  and  Defense  Contract  Management  Agency 
(DCMA)  are  working  with  Bell  Helicopter  Textron  Inc.  to  implement 
improved  rates  and  overhead  cost  control  processes. 

Cost  Estimating  Area 

Significant  contractor  rates 
increased 

Revised  estimate. 

Production  labor  &  materials  more  Revised  estimate  and  initiated  cost  reduction  efforts. 


than  planned 

Materials  more  expensive  than 
estimated 

Revised  estimate  based  on  EMD  actuals  and  independent  review  by 
NAVAIR  and  CAIG. 

Too  aggressive  learning  curve 
assumed 

Revised  production  estimate  using  flatter  curve. 

Funding  Area 

Unfunded  training  requirements 

Added  funds  to  cover  these  costs. 

Schedule  Area 

Insufficient  testing  time 

Schedule  extended. 

Insufficient  logistics  validation 
period 

Schedule  extended. 

SOURCES:  "H-1  UPGRADES  Selected  Acquisition  Reports"  and  Acquisition  Decision  Memoranda 


Multiple  Nunn-McCurdy  Breaches  91 


technological  issues.  The  actions  taken  at  the  second  breach  are  shown  by  shaded  rows. 
The  actions  taken  for  this  program  were  not  as  clearly  called  out  in  the  SARs  as  they 
were  for  the  other  programs. 

Key  Takeaways  from  H-1  Upgrades  Review 

The  H-1  Upgrades  program  office  faced  schedule  breaches  and  RDT&E  cost  growth 
throughout  its  EMD  phase.  One  main  reason  was  a  persistent  problem  with  contractor 
execution  and  increasing  rates.  The  program  also  had  late  emerging  technical  problems 
caused  by  the  recycle  of  designs.  In  addition,  the  cost  estimating  assumptions  were 
overly  optimistic  causing  frequent  rebaselining.  Finally,  external  events  (i.e.,  combat 
losses  of  aircraft  frames)  resulted  in  a  change  to  the  acquisition  plan  whereas  new 
airframes  were  purchased  rather  than  refurbishing  older  ones  (which  were  no  longer 
available). 

Joint  Primary  Aircraft  Training  System 

JPATS  is  an  Air  Force  and  Navy  program  that  is  meant  to  “replace  USAF’s  T-37B 
aircraft,  USN’s  T-34C  aircraft,  and  the  associated  Ground  Based  Training  Systems 
(GBTS).  The  aircraft  and  GBTS  are  being  used  to  train  entry-level  students  in  the 
fundamentals  of  flying  so  they  can  transition  into  advanced  training  tracks  leading 
to  rated  qualification.  The  program  represents  a  systems  approach  to  aviator  training 
requiring  the  purchase  of  air  vehicles  (747  production  units),  aircrew  training  devices 
(126),  associated  ground  based  training  devices,  an  integrated  training  information 
management  system  (TIMS),  instructional  courseware,  as  well  as  the  entire  logistics 
and  sustainment  of  the  training  system  which  includes  contractor  logistics  support 
(CFS).”36  The  program  was  awarded  its  Milestone  II  in  August  1995  and  its  Milestone 
III  in  December  2001. 

Unlike  the  previous  programs  discussed  in  this  section,  JPATS  is  not  a  multiple 
Nunn-McCurdy  breach  program.  As  shown  in  Figure  2.11,  the  program  reported 
Nunn-McCurdy  breaches  in  both  2005  and  2006;  however,  the  breaches  in  2005  were 
not  resolved  administratively  by  2006,  so  we  consider  these  as  only  one  breach.  This 
trend  frequently  happens  when  breaches  are  reported  because  it  takes  a  considerable 
time  for  programs  to  go  through  the  Nunn-McCurdy  process  and  for  revisions  to  be 
made  to  the  program’s  path  so  that  future  cost  growth  can  be  avoided.  This  program 
may  also  have  been  caught  up  in  the  change  in  Nunn-McCurdy  reporting  require¬ 
ments  during  the  time  period  of  the  breach.  This  change,  which  included  additional 
reporting  of  unit  cost  growth  against  the  original  baseline,  negatively  affected  pro¬ 
grams  that  would  not  have  otherwise  breached.  We  also  cannot  rule  out  the  possibil¬ 
ity  that  the  program  was  avoiding  breaching  by  rebaselining  four  times  from  1998 


36  “jpATS  Selected  Acquisition  Report  (SAR),”  Defense  Acquisition  Management  Information  Retrieval 
(DAMIR),  December  31,  2010,  p.  3. 
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Figure  2.11 

JPATS  Nunn-McCurdy  Breach  History 


Current  baseline 
Original  baseline 


1998  1999  2000  2001  2002  2003  20041 20051 2006  2007  2008  2009  2010 

PAUC 

Apuc 


New  APB  or  rebaseline 

Critical  breach 

Significant  breach 

'M. 

Not  tracked 

SOURCE:  "JPATS  Selected  Acquisition  Reports"  (1998  through  2010). 
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through  2004.  Part  of  the  rationale  for  the  change  in  legislation  in  2006  was  to  shed 
light  on  cost  growth  that  may  not  have  been  as  visible  because  of  frequent  rebaselining. 

Figure  2.12  tracks  the  program’s  percentage  change  in  the  PAUC  and  APUC 
against  both  the  current  and  original  baselines  as  available  since  December  1997.  As 
can  be  seen  in  the  chart,  the  PAUC  and  APUC  have  been  relatively  steady  over  time 


Figure  2.12 

JPATS  PAUC  and  APUC  History 


SOURCE:  "JPATS  Selected  Acquisition  Reports"  (1997  through  2010). 


RAND  MG1 1 71/5-2. 12 
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except  in  2005/2006  when  both  rose  significantly  when  the  original  baseline  was 
introduced: 

•  The  PAUC  against  the  original  baseline  increased  44  percent;  and 

•  The  APUC  against  the  original  baseline  increased  55  percent. 

From  1997  to  2004,  the  program  experienced  minimal  APB  breaches  and  no 
Nunn-McCurdy  breaches.  JPATS  had  an  APB  procurement  cost  breach  in  1998  and 
an  APB  schedule  breach  in  2002.  It  did  not  have  any  APB  breaches  in  2003  or  2004. 
Until  the  2005  significant  Nunn-McCurdy  breach  of  both  the  APUB  and  PAUC 
against  the  original  baseline,  the  program  appears  to  have  been  controlling  cost  even 
with  multiple  instances  of  technical  setbacks. 

Difficulties  experienced  by  the  program  before  the  2005  breach  include  a  delay  in 
awarding  the  EMD  contract  in  1995  at  Milestone  II  because  two  protests  were  bled  by 
Rockwell  and  Cessna  when  Raytheon  was  awarded  the  contract.  The  GAO  denied  all 
allegations  and  the  first  production  lot  option  was  exercised  in  February  1996.  In  the 
next  couple  of  years  following  Milestone  II,  assembly  of  the  first  test  aircraft  (T-l)  was 
behind  schedule.  The  delay  was  caused  by  a  late  delivery  of  both  the  tools  and  com¬ 
puter-aided  manufacturing  software.  There  was  also  an  industrywide  shortage  of  expe¬ 
rienced  workers  who  were  working  on  this  test  aircraft.  These  complications  caused  a 
seven-week  delay  in  the  T-l  aircraft. 

In  FY  1999,  the  program’s  funding  level  would  not  permit  the  execution  of  the 
FY  1999  contract  options  as  planned  to  install  TIMS  at  all  seven  Air  Education  and 
Training  Command  (AETC)  pilot  training  bases.  The  funding  complications  in  FY 
1999  caused  some  additional  schedule  delays  for  the  program  along  with  the  fact  that 
the  TIMS  system  was  difficult  to  develop. 

In  January  1999,  the  program  experienced  several  technical  issues.  The  P-2  air¬ 
craft  engine  malfunctioned,  which  forced  the  program  to  ground  all  aircraft.  In  addi¬ 
tion,  in  February  1999,  the  rear  fuselage  also  needed  structural  modifications.  Finally, 
the  program  experienced  problems  with  the  environmental  control  system,  the  engine’s 
automatic  airstart  system,  and  the  aircraft’s  empennage  structure  system.  This  combi¬ 
nation  of  technical  problems  caused  the  program  office  to  prepare  a  new  baseline  for 
the  program  in  1999. 

During  2002,  the  aircraft  production  contract  was  changed  from  award  fee  to  a 
performance  incentive  fee  structure.  According  to  the  SAR, 

The  reason  for  the  change  is  that  the  old  contract  delivery  schedule  did  not  provide 
sufficient  margin  between  aircraft  availability  and  aircraft  requirements  (training 
and  retrofit)  in  the  December  2002  through  February  2003  time  frame.  The  SPO 
chose  to  incentivize  early  deliveries  to  ensure  a  sufficient  margin  of  aircraft  avail¬ 
ability.  PT-75  and  on  were  delivered  ahead  of  contract  schedule  as  a  direct  result 
of  this  contract  incentive.  An  additional  benefit  of  the  accelerated  aircraft  delivery 
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schedule  was  that  a  greater  number  of  aircraft  with  the  upgraded  environmental 
control  system  (ECS)  were  involved  in  the  MOT&E(S).37 

The  program  experienced  more  technological  setbacks  in  both  2004  and  2005. 
In  April  2004,  the  T-6A  crashed,  killing  two  instructor  pilots.  A  detailed  investigation 
was  done  to  understand  the  cause  of  the  crash.  Then,  in  2005,  the  Navy  reported  an 
in-flight  engine  failure.  This  led  to  another  investigation.  JPATS  engineers  and  Pratt  & 
Whitney  concluded  that  all  engines  over  2,000  hours  were  at  risk  of  turbine  blade  fail¬ 
ure,  forcing  both  the  Air  Force  and  Navy  to  stop  using  aircraft  with  over  2,000  flight 
hours  and  make  repairs. 

Also  in  2005,  the  program  had  to  start  reporting  unit  cost  against  the  program’s 
“original”  APB,  i.e.,  the  APB  established  at  Milestone  B  (previously  Milestone  II). 
Because  of  this  new  requirement,  the  program  had  to  report  an  increase  in  the  PAUC 
of  37  percent  and  47  percent  in  the  APUC.  The  program  reported  the  following  as  the 
causes  for  the  breaches: 

The  PAUC  increase  from  MS  II  to  MS  III  was  caused  by  an  increase  in  cost  due 
to  foreign  sales  not  materializing,  increased  material  costs,  flattening  of  the  learn¬ 
ing  curve  and  a  decrease  in  quantity.  The  PAUC  increase  from  MS  III  to  present  is 
a  result  of  increased  estimate  of  unit  flyaway  cost,  redetermination  of  lead  service 
change  cost,  the  decision  to  procure  all  required  Navy  program  related  spares  and 
other  minor  changes  in  support,  engineering  changes  and  mission  support.  The 
APUC  increase  from  MS  II  to  MS  III  was  caused  by  an  increase  in  cost  due  to 
foreign  sales  not  materializing,  increased  material  costs,  flattening  of  the  learning 
curve  and  a  decrease  in  quantity.  The  APUC  increase  from  MS  III  to  present  is  also 
due  to  an  increased  estimate  of  unit  flyaway  costs,  redetermination  of  lead  service 
change  cost,  the  decision  to  procure  all  required  Navy  program  related  spares  and 
other  minor  changes  in  support,  engineering  changes  and  mission  support.38 

The  program  reported  even  greater  percentage  changes  in  the  PAUC  and  APUC 
against  the  original  baseline  in  2006.  The  PAUC  increased  44  percent  against  the 
original  baseline  and  the  APUC  increased  nearly  55  percent  against  the  same  baseline 
triggering  a  significant  Nunn-McCurdy  breach  for  the  PAUC  metric  and  a  critical 
Nunn-McCurdy  breach  for  the  APUC  metric.  The  program  office  used  a  similar  expla¬ 
nation  for  the  breaches  as  in  2005.  Table  2.7  consolidates  the  various  problems  relevant 
to  the  breach  that  the  program  has  experienced  since  1997.  It  then  provides  any  issues 
and  actions  taken  to  correct  the  problems  moving  forward. 


37  “JPATS  Selected  Acquisition  Report  (SAR),”  December  31,  2002,  p.  6. 

38  “JPATS  Selected  Acquisition  Report  (SAR),”  Defense  Acquisition  Management  Information  Retrieval 
(DAMIR),  December  31,  2005,  p.  29. 
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Table  2.7 
JPATS  SAR  Issues 


Area 

issues 

Actions 

Technical  risk  and  maturity 

Technical  problems  with  engine, 
landing  gear,  and  airframe 

Redesign  of  affected  components 

Scope  and  requirements 

Development  and  acquisition 
strategy 

Protest  on  contract  award 

Delay  of  program  start,  but  both 
protests  were  denied 

Lack  of  schedule  incentive  for 
contractor 

Change  from  award  to  incentive 
fee  contract 

Performance  management 

Cost  estimating 

Underestimated  cost 

Revised  manufacturing  assumptions 

Funding 

Navy  deferred  procurement  in 

FY  2002 

Increase  of  unit  costs 

Schedule 

Shortage  of  skilled  workers 

Schedule  delayed;  contract  changed 
to  incentive  fee 

SOURCES:  "JPATS  Selected  Acquisition  Reports"  and  Acquisition  Decision  Memoranda. 


Key  Takeaways  from  JPATS  Review 

JPATS  experienced  numerous  technical  issues  during  production  that  required  fixes. 
The  program  identified  the  causes  of  these  issues  and  implemented  solutions  using 
redesign  and  active  maintenance  without  causing  major  delays  in  the  program’s  sched¬ 
ule.  Actions  taken  also  minimized  cost  growth.  Toward  the  end  of  production,  the 
Navy  decided  to  cut  quantities  by  20.  This  also  resulted  in  a  cut  in  funding  by  Con¬ 
gress  in  FY  2011.  The  program  also  anticipated  that  foreign  sales  would  help  to  offset 
the  price.  These  sales  materialized  only  late  in  the  program  and  did  not  help  to  reduce 
costs  stemming  from  cuts  to  the  U.S.  quantities.  However,  the  program  has  not  expe¬ 
rienced  another  Nunn-McCurdy  breach  because  of  the  quantity  reduction. 

It  is  possible  that  this  program  did  not  experience  a  repeat  Nunn-McCurdy 
breach  because  it  was  able  to  deal  with  its  technological  problems  and  was  four  years 
into  production  at  the  time  of  the  first  breach.  Also,  this  program  was  one  of  those 
that  seemed  to  breach  once  the  Nunn-McCurdy  law  was  changed  in  2006  to  require 
reporting  against  the  original  baseline.  Had  the  law  change  not  occurred,  it  is  possible 
this  program  would  not  have  breached  even  once. 
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Observations 

The  change  in  the  Nunn-McCurdy  law  in  2006  profoundly  affected  the  way  breaches 
are  reported.  Before  2006,  programs  could  rebaseline.  So  if  the  cost  growth  was  not 
rapid  or  abrupt,  breaches  against  the  original  baseline  could  be  masked.  Now,  because 
programs  report  against  the  original  baseline,  many  more  appear  to  breach  now  than 
in  the  past.  For  example,  Congressional  Research  Service  noted39  that  11  programs 
were  reclassified  as  having  breaches  after  the  law  change.  Thus,  it  is  very  difficult  for 
us  to  state  whether  breaches  have  become  more  frequent  recently.  The  breaches  are  cer¬ 
tainly  recognized  more  often  now. 

Our  answers  to  the  original  questions  of  this  research  follow. 

Are  programs  that  have  a  Nunn-McCurdy  breach  more  likely  to  breach 
again?  The  evidence  we  have  gathered  does  not  suggest  that  programs  that  have  one 
breach  are  more  likely  to  breach  again.  However,  our  sample  size  is  small,  and  the  2006 
law  change  makes  any  conclusive  view  impossible. 

What  can  we  learn  from  the  cost  growth  trends  of  those  programs  that  have 
had  multiple  Nunn-McCurdy  breaches?  We  observed  that  most  of  the  programs 
with  multiple  breaches  tend  to  have  very  rapid  cost  growth  displayed  in  the  SARs. 
They  jump  directly  into  a  critical  breach  between  years  or  rapidly  move  from  signifi¬ 
cant  to  critical  breach.  There  seems  to  be  no  advanced  warning  from  the  cost  growth 
that  indicates  a  potential  breach.  Also,  the  APUC  and  PAUC  measures  are  highly  cor¬ 
related  for  most  programs.  We  explored  growth  in  the  cost  variance  categories  and  did 
not  find  anything  of  note. 

What  can  we  learn  from  the  management  actions  taken  on  programs  with 
multiple  breaches?  It  is  very  difficult  to  answer  this  question  with  the  limited  program 
information  we  had  available.  After  interpreting  the  SARs  and  ADMs,  we  attempted 
to  classify  the  major  actions  taken  at  the  first  breach  and  how  they  did  or  did  not  affect 
the  second  breach.  We  classified  the  actions  taken  at  the  first  breach  as  follows  (in 
accordance  with  the  PARCA  office  recommendations): 

•  Did  not  recognize  the  problem — the  issues  that  led  to  the  second  breach  were  not 
understood  at  the  time  of  the  initial  breach. 

•  Recognized  the  problem  but  took  no  corrective  action — the  issue  related  to  the 
second  breach  was  identified,  but  no  specific  action  was  taken  to  prevent  it. 

•  Recognized  the  problem  and  did  not  take  the  appropriate  corrective  action — the 
problem  that  led  to  the  second  breach  was  recognized,  but  the  action  taken  did 
not  avert  it  or  was  ineffective. 


39  Schwartz,  2011. 
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•  Recognized  the  problem  and  took  action,  but  did  not  fully  understand  impli¬ 
cations — the  issues  leading  to  the  second  breach  were  recognized,  but  the  full 
implications  of  the  problem  were  not  understood. 

•  Recognized  the  problem  and  took  appropriate  action. 

•  Issues  took  more  time  to  resolve  than  time  to  rebaseline — the  corrective  actions 
were  difficult  to  implement  and  the  second  breach  occurred  before  the  initial 
actions  were  fully  implemented. 

In  Table  2.8,  we  attempt  to  classify  the  issues  and  categorization  of  the  actions 
taken  at  the  first  breach  for  the  four  programs  we  explored.  Again,  this  table  is  a  highly 
speculative  and  subjective  assessment.  It  is  interesting  to  note  that  we  found  no  clear- 
cut  instances  where  no  action  was  taken.  For  all  three  of  the  programs  with  multiple 
breaches,  resolving  the  technical  maturity  issues  took  time  and  was  still  an  issue  at  the 
second  breach.  For  the  SBIRS  and  H-l  Upgrades  programs,  there  was  focus  on  rebase- 

Table  2.8 

Issues  Versus  Actions  Taken  at  First  Breach 


SBIRS 

C-130  AMP 

H-1  Upgrades 

JPATS 

Did  not  recognize 
the  problem 

Schedule  plan  was 
not  realistic 

Recognized  the 
problem  but  took 
no  corrective  action 

Recognized  the 
problem  and 
did  not  take 
the  appropriate 
corrective  action 
or  actions  were 
ineffective 

Rebaseline  and 
improved  EVM  did 
not  limit  further  costs 
growth 

Rebaseline  and 
improved  EVM  did 
not  limit  further 
costs  growth 

Recognized  the 
problem  and  took 
action,  but  did  not 
fully  understand 
implications 

Change  of  acquisition 
strategy  from  TSPR 
to  more  government 
control 

Increased  cost  and 
time  led  to  quantity 
cuts  from  partners, 
change  from 
sole  source  to 
competitive 
production  also 
added  cost 

Increased  cost  and 
time  led  to  quantity 
cuts  from  partners 

Recognized  the 
problem  and  took 
appropriate  action 

Insufficient  technical 
and  program 
manager  resources 

ICE  needed 

Restructured 

contract, 

improved  technical 
maturity 

Issues  took  more 
time  to  resolve 
than  time  to 
rebaseline 

Improved  technical 
maturity, 

realistic  cost  baseline, 
insufficient  technical 
and  program 
manager  resources 

Improved  technical 
maturity 

Improved 
technical  maturity 
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lining  and  fixing  the  EVM  system  so  that  it  accurately  reflected  the  program  status. 
However,  the  new  cost  baseline  was  unrealistic  in  both  cases  as  the  technical  issues 
and  contractor  performance  continued  to  challenge  both  programs.  Some  program 
changes  had  implications  that  were  not  fully  understood.  The  most  striking  of  these 
were  the  two  acquisition  strategy  changes  for  SBIRS  High  and  C-130  AMP.  The  other 
noteworthy  observation  from  the  table  is  the  vulnerability  of  joint  or  international  pro¬ 
grams  to  quantity  cuts  when  costs  increase — leading  to  the  so-called  “unit  cost  death 
spiral”  (cost  increases  lead  to  quantity  cuts,  which  leads  to  further  cost  increases  .  .  . 
and  so  on). 

In  the  discussion  below,  we  address  the  last  question,  are  there  common  issues 
that  were  missed  at  the  first  breach?  In  Table  2.9,  we  summarize  the  common  issues 
observed  across  the  four  programs  examined.  Note  that  all  programs  had  schedule 
and  technical  issues,  even  JPATS,  which  did  not  breach  multiple  times.  The  two  char¬ 
acteristics  common  to  the  programs  with  multiple  breaches  were  that  they  had  sig¬ 
nificant  quantity  variances  and  immature  cost  estimates.  It  should  be  noted  that  the 
quantity  issue  is  more  associative  than  causal.  Another  observation  that  can  be  made 
from  Table  2.9  is  that  two  of  the  programs  with  multiple  breaches  had  major  changes 
to  their  acquisition  strategy  and  direct  congressional  influence  in  the  program.  These 
observations  suggest  that  further  research  could  be  done  on  a  more  complete  set  of 
programs  looking  for  common  characteristics  and  trying  to  quantify  how  they  might 
influence  the  likelihood  of  multiple  breaches.  For  example,  further  research  might 
show  whether  programs  with  significant  quantity  changes  are  more  prone  to  another 
breach  or  whether  quantity  changes  are  just  a  symptom  of  other  program  problems. 

The  common  issues  listed  in  Table  3.9  have  some  similarities  to  other  root  causes 
for  other  DoD  programs  identified  in  recent  research  done  by  RAND  (for  PARCA). 


Table  2.9 
Common  Issues 


Issue 

SIBRS  High 

C-130  AMP 

HI  Upgrades 

JPATS 

Significant  quantity  variances 

X 

X 

Xa 

Major  change  in  acquisition  strategy 

X 

X 

Protests 

X 

X 

Technical  problems 

X 

X 

X 

X 

Congressional  redirection 

X 

X 

Cost-estimating  not  mature  until  late 

X 

X 

X 

X 

Schedule  delays 

X 

X 

X 

X 

Joint/international 

X 

X 

a  Combat  losses  of  aircraft  frames  resulted  in  a  change  to  the  acquisition  plan  in  which  new  airframes 
were  purchased  rather  than  refurbishing  older  ones,  thus  raising  the  cost. 
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Blickstein  et  al.  summarize  root  causes  for  several  programs  that  have  had  Nunn- 
McCurdy  breaches  (i.e.,  Joint  Strike  Fighter,  Excalibur,  Wideband  Global  Satellite, 
Longbow  Apache  DDG-1000,  and  JTRS  GMR).40  Several  of  the  root  causes  identi¬ 
fied  are  similar  to  those  we  have  observed  in  this  study.  The  common  root  causes  are 
as  follows: 

•  changes  (up  or  down)  in  procurement  quantities 

•  underestimation  of  costs/overoptimistic  cost  estimates 

•  immature  technologies  resulting  in  later  technical  problems 

•  ineffective  contract  incentives. 


Condusions 

This  chapter  has  explored  three  issues  related  to  programs  that  have  multiple  Nunn- 
McCurdy  breaches.  We  found  that  programs  that  breached  once  are  not  more  likely 
to  breach  a  second  time.  The  2006  change  in  the  Nunn-McCurdy  law  has  increased 
awareness  of  programs  that  breach  above  their  initial  baselines.  We  have  found  no 
obvious  cost  growth  trends  that  would  suggest  that  a  program  might  breach  more 
than  once.  In  terms  of  actions  taken  at  the  first  breach,  those  that  breached  more  than 
once  had  technical  issues  that  were  not  resolved  by  corrective  actions  taken  at  the  first 
breach.  Finally,  we  did  find  some  common  characteristics  among  those  programs  with 
multiple  breaches.  But  because  our  sample  was  small,  the  results  are  not  definitive,  and 
further  research  might  refine  this  view. 


40  Irv  Blickstein,  Michael  Boito,  Jeffrey  A.  Drezner,  James  Dryden,  Kenneth  Horn,  James  G.  Kallimani,  Martin 
C.  Libicki,  Megan  McKernan,  Roger  C.  Molander,  Charles  Nemfakos,  Chad  J.  R.  Ohlandt,  Caroline  Reilly, 
Rena  Rudavsky,  Jerry  M.  Sollinger,  Katharine  Watkins  Webb,  and  Carolyn  Wong,  Root  Cause  Analyses  of  Nunn- 
McCurdy  Breaches,  Volume  1:  Zumwalt-Class  Destroyer,  Joint  Strike  Fighter,  Longbow  Apache,  and  Wideband  Gobal 
Satellite,  Santa  Monica,  Calif.:  RAND  Corporation,  MG-1171/1,  2011. 
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Nunn-McCurdy  Legislation 


The  Nunn-McCurdy  Act,  whose  purpose  is  to  help  control  cost  growth  in  MDAPs, 
requires  that  the  DoD  report  unit  costs  for  major  weapons  systems  to  Congress.  This 
legislation  was  originally  signed  into  law  in  the  early  1980s  and  has  undergone  a  variety 
of  changes  over  30  years.  Nunn-McCurdy  legislation  established  thresholds  as  a  way  of 
monitoring  cost  growth.  When  cost  growth  surpasses  the  thresholds  established  in  the 
legislation,  a  process  is  set  in  motion  whereby  the  program  office  and  other  parties  in 
the  DoD  must  notify  Congress  of  the  growth  and  reasons  behind  it. 


Original  Nunn-McCurdy  Legislation 

In  1981,  Senator  Samuel  Nunn  and  Congressman  David  McCurdy  introduced  the 
Nunn-McCurdy  amendment1  to  the  Department  of  Defense  Authorization  Act  of 
1982. 2  The  purpose  of  the  amendment  was  to  establish  congressional  oversight  of 
defense  weapon  system  acquisition  programs  that  experience  cost  growth  above  limits 
specified  in  the  amendment.  The  Nunn-McCurdy  amendment  defined  two  types  of 
unit  cost:  total  PAUC,  which  is  the  sum  of  development  funding  and  procurement 
funding  divided  and  military  construction  by  units  procured;  and  APUC,  which  is  the 
procurement  funding  divided  by  units  procured.  Cost  growth  of  a  weapon  system  was 
measured  by  how  much  the  unit  costs  in  1982  exceeded  the  same  respective  unit  costs 
in  the  weapon  system’s  SAR  dated  March  31,  1981.  Hence,  the  amendment  applied 
only  to  those  major  weapon  systems  with  March  31,  1981,  SARs. 

The  original  amendment  required  that  the  Secretary  of  Defense  (SECDEF)  notify 
Congress  when  a  major  weapon  system  unit  cost  growth  exceeded  15  percent.  If  unit 
cost  growth  exceeded  25  percent,  the  program  was  assumed  terminated  unless  the  Sec¬ 
retary  of  Defense  submitted  specific  written  certifications  to  Congress  within  60  days 
of  making  the  cost  growth  determination.  These  certifications  survive  in  current  law. 

1  The  Nunn-McCurdy  amendment  is  also  known  as  the  Nunn-McCurdy  provision.  See  Nunn-McCurdy 
Amendment,  Department  of  Defense  Authorization  Act,  1982,  Report  No.  97-311,  November  3,  1981. 

2  Public  Law  97-86,  National  Defense  Authorization  Act  of  1982,  December  29,  1981. 
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Congress  made  the  provisions  of  the  Nunn-McCurdy  amendment  permanent  in 
the  1983  Authorization  Act3  by  requiring  that  the  secretary  of  each  military  depart¬ 
ment  establish  a  baseline  description  of  each  major  weapon  system  acquisition  program 
under  the  jurisdiction  of  the  secretary.  The  baseline  description  was  to  include  a  base¬ 
line  estimate  of  the  program  cost.  The  permanent  Nunn-McCurdy  provisions  mea¬ 
sured  unit  cost  growth  by  comparing  the  current  unit  costs  against  the  same  respective 
unit  costs  in  the  baseline  estimate.  The  cost  thresholds  for  notifying  Congress  and 
for  program  termination  presumptions  in  the  original  Nunn-McCurdy  amendment 
remained  unchanged  in  the  1983  Authorization  Act  but  have  subsequently  changed. 


Changes  to  Nunn-McCurdy  Legislation  Since  1982 

Since  the  original  Nunn-McCurdy  legislation  was  enacted,  Nunn-McCurdy  legisla¬ 
tion  has  evolved  with  significant  changes  to  the  legislation  in  both  2006  and  2009. 
Other  changes  were  relatively  minor  in  comparison  and  included  changes  to  previ¬ 
ously  established  thresholds,  definitions  of  unit  cost  measures,  Nunn-McCurdy  pro¬ 
cess  timelines/deadlines,  and  documentation  requirements.  See  Figure  A.l  for  more 
details  on  the  changes  over  time: 

The  FY  2006  National  Defense  Authorization  Act  (NDAA)  (P.L.  109-163) 
included  a  major  addition  to  the  Nunn-McCurdy  legislation  that  affected  MDAPs  and 
the  management  of  those  programs.  Congress  mandated  that  cost  growth  be  measured 
against  the  current  baseline  estimate  and  the  original  baseline  estimate.  In  fiscal  year 
2006: 

Congress  added  the  original  baseline  estimate  as  a  benchmark  against  which  to 
measure  cost  growth.  The  original  baseline  estimate  is  defined  as  the  baseline 
description  prepared  before  the  program  enters  development,  or  at  program  ini¬ 
tiation,  whichever  is  later,  without  adjustment  or  revision.  By  adding  the  origi¬ 
nal  baseline  estimate  as  a  benchmark  against  which  to  measure  cost  growth,  and 
by  restricting  the  circumstances  in  which  an  original  baseline  estimate  may  be 
revised,  DOD  can  no  longer  avoid  Nunn-McCurdy  breaches  by  simply  revising 
a  program’s  baseline  estimate.  While  DOD  acquisition  policy  still  allows  current 
baseline  estimates  to  be  revised,  the  policy  was  modified  in  2008  to  limit  the  cir¬ 
cumstances  under  which  this  may  be  done.4 

As  a  result  of  this  legislation,  more  programs  had  Nunn-McCurdy  breaches  than 
otherwise  would  have  been  the  case.  “According  to  DOD,  11  programs  that  did  not 
have  a  Nunn-McCurdy  breach  prior  to  the  new  FY2006  requirements  were  recatego- 


3  Public  Law  97-252,  National  Defense  Authorization  Act  of  1983,  September  8,  1982. 

4  Government  Accountability  Office,  2011a,  pp.  3-4. 


Figure  A.1 

Key  Events  in  History  of  Nunn-McCurdy  Legislation 


DoD  Authorization 
Act  of  1982  (May  1981): 
Required  DoD  to 
notify  Congress  if  cost 
growth  exceeded 
certain  thresholds 
(only  required  for  cost 
overruns  in  FY  1982); 
defined  thresholds  for 
PAUC/PUC 


FY  1985  DoD 
Authorization  Act 
(Oct  1984):  Changed 
definition  of  PUC; 
changed  definition 
of  "major"  contract; 
Nunn-McCurdy  does 
not  apply  to  programs 
nearing  completion; 
time  line  extended 
for  SAR  submission 


FY  1993  NDAA 
(Oct  1992):  Slightly 
modified  Nunn-McCurdy 
thresholds  and  time  line 
for  notification  of 
breach  to  Congress 


FY  2007  NDAA  (Oct  2006): 
Included  original  baseline 
estimate  as  a  standard 
against  which  to  measure 
cost  growth;  introduced 
"significant"  and  "critical" 
terms:  11  programs  that 
did  not  have  a  breach  had 
significant  breaches  as 
a  result  of  this  legislation 


Weapon  System 
Acquisition  Reform 
Act  of  2009  (May  2009): 
Added  an  analysis  for 
critical  cost  growth  and 
completing  the  program/ 
alternatives;  program 
is  terminated  unless 
certification  of  need  by 
SECDEF;  most  recent 
milestone  is  revoked 
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to  all  major 

to  rescind  the 

"baseline  selected  acquisition 
report;"  changed  reporting 
requirements  for  program 
manager  and  time  line  for 
submitting  Congressional 
notification  of  breach 
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reporting  requirements 
in  SAR;  penalty  for 
changed  for  failing  to 
submit  SAR  at  time  of 
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base  year  dollars 
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approval  for 
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change  in 
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rized  as  having  significant  breaches  as  a  result  of  the  legislation’s  new  original  baseline.”5 
This  change  in  legislation  is  particularly  relevant  to  this  section  of  the  report  on  repeat 
Nunn-McCurdy  programs.  Some  of  the  repeat  Nunn-McCurdy  breaches  can  be  attrib¬ 
uted  in  part  to  changes  in  legislation.  In  addition  to  legislative  changes,  the  GAO  also 
attributes  some  of  the  repeat  breaches  to  changes  in  presidential  administration.6 

Approximately  three  years  after  the  2006  legislation,  the  Weapon  System  Acquisi¬ 
tion  Reform  Act  (WSARA)  of 2009  (RL.  111-23)  was  passed.  WSARA  revised  Nunn- 
McCurdy  laws  to  include  a  more  complicated  process  and  established  a  new  office 
to  examine  the  causes  of  Nunn-McCurdy  breaches  and  related  issues.  Specifically, 
WSARA  required  the  following  for  programs  that  the  USD  (AT&L)  believed  should 
not  be  terminated: 

•  Additional  certification  to  Congress  is  required,  stating  that  the  program  is  higher 
priority  than  programs  whose  funding  must  be  cut  to  cover  the  cost  growth  of 
current  program. 

•  Revocation  of  most  of  the  recent  milestone  approval  is  required  and  no  new  con¬ 
tracts  can  be  awarded  without  new  milestone  approval  or  MDA  approval. 

•  Analysis  should  be  conducted  to  determine  the  root  cause  of  cost  growth. 

•  Program  must  be  restructured  to  address  root  causes  of  cost  growth. 

•  Failure  to  certify  to  Congress  the  results  of  the  above  findings  results  in  program 
termination.7 

The  WSARA  changes  may  be  the  most  pivotal  since  the  Nunn-McCurdy  leg¬ 
islation  was  enacted  in  1982.  In  fact,  they  may  have  been  too  extensive  as  Congress 
backtracked  on  some  of  the  requirements  from  WSARA  regarding  Nunn-McCurdy 
reporting  in  Section  801  of  Title  VIII  of  the  FY  2012  legislation  because  of  concerns 
about  the  burden  (or  costs)  of  compliance: 

The  committee  recommends  a  provision  that  would  allow  the  waiver  of  certain 
requirements  applicable  to  programs  that  experience  critical  Nunn-McCurdy 
breaches  as  a  result  of  steep  growth  in  unit  costs,  in  cases  where  such  cost  growth 
is  attributable  entirely  (or  almost  entirely)  to  changes  in  the  number  of  units  to 
be  purchased.  The  provision  recommended  by  the  committee  includes  strict  stan¬ 
dards  to  ensure  that  all  Nunn-McCurdy  requirements  remain  applicable  in  any 
case  where  poor  program  management  or  performance  contributes  to  the  increase 
in  unit  costs.8 


5  Schwartz,  2011,  p.  10. 

6  Government  Accountability  Office,  2011a,  p.  2. 

7  Schwartz,  2011,  p.  10. 

8  National  Defense  Authorization  Act  for  Fiscal  Year  2012,  112th  Congress,  1st  Session  Report,  June  22,  2011, 
p.  135. 


Nunn-McCurdy  Legislation  105 


Current  Nunn-McCurdy  Process 

The  current  Nunn-McCurdy  process  is  largely  the  same  as  the  original  outside  the 
major  additions  in  both  2006  and  2009.  If  a  program  breaches  the  Nunn-McCurdy 
thresholds,  then  the  program  undergoes  considerable  scrutiny  by  Congress  and  the 
DoD  in  which  the  program  office  must  provide  Congress  with  reasons  for  cost  growth 
and  plans  for  how  to  avoid  it  in  the  future.  Currently,  there  are  two  unit  cost  criteria 
that  are  considered  for  the  thresholds.  The  first  is  PAUC9  and  the  second  is  APUC.10 
Both  are  required  to  be  reported  in  base  year  dollars  to  take  into  account  inflation. 
Both  of  the  current  estimates* 11  of  these  unit  costs  are  then  compared  to  both  the  cur¬ 
rent  baseline  estimate12  and  to  the  original  baseline  estimate.13  The  law  requires  spe¬ 
cific  actions  and  reporting  if  a  program  breaches  the  unit  cost  thresholds.  A  significant 
breach  occurs  when  the  current  baseline  estimate  is  breached  by  15  percent  or  the 
original  baseline  estimate  is  breached  by  30  percent.  A  critical  breach  occurs  when  the 
current  baseline  estimate  is  breached  by  25  percent  or  the  original  baseline  estimate  is 
breached  by  50  percent. 

If  a  program  has  a  significant  Nunn-McCurdy  breach,  the  appropriate  service 
secretary  must  notify  Congress  within  45  days  of  the  unit  cost  report.  This  usually 
takes  the  form  of  a  “program  deviation  report.”  The  DoD  then  submits  an  SAR  with 
required  unit  cost  breach  information  (this  may  be  a  quarterly  SAR  or  can  be  included 
in  the  annual  SAR). 

9  PAUC  =  [Total  Development  $  +  Procurement  $  +  Construction  $]  /  Total  program  quantity. 

10  APUC  =  Total  Procurement  $  /  Procurement  quantity. 

11  Latest  estimate  of  approved  program. 

12  Currently  approved  APB. 

13  APB  approved  at  Milestone  B  or  program  initiation,  whichever  occurs  later. 
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List  of  Programs  with  at  Least  One  Nunn-McCurdy  Breach 
(1997-2011) 


Advanced  Anti-Tank  Weapon  System — Medium  (Javelin) 

Advanced  Extremely  High  Frequency  (AEHF)  Satellite 
Advanced  Field  Artillery  Tactical  Data  System  (AFATDS) 

Advanced  SEAL  Delivery  System  (ASDS) 

Advanced  Threat  Infrared  Countermeasure/Common  Missile  Warning  System 
(ATIRCM/CMWS) 

AH-64D  LONGBOW  APACHE 
AH-64E  Apache  Remanufacture 
Armed  Reconnaissance  Helicopter  (ARH) 

Army  Tactical  Missile  System  (TACMS)/BAT 

B-1B  Conventional  Mission  Upgrade  Program  (CMUP) 

C-130  Avionics  Modernization  Program  (AMP) 

C-130J  Hercules  Transport  Aircraft 
C-27J  Spartan 

C-5  Reliability  Enhancement  and  Reengineering  Program  (RERP) 

CH-47F  Improved  Cargo  Helicopter  (CH-47F) 

Chemical  Demilitarization  Program 

Chemical  Demilitarization— Assembled  Chemical  Weapons  Alternatives  (Chem 
Demil-ACWA) 

Chemical  Demilitarization— Chemical  Materials  Agency  Newport  (Chem  Demil— 
CMA  Newport) 

Chemical  Demilitarization— U.S.  Army  Chemical  Materials  Agency  (Chem 
Demil-CMA) 

Comanche  Reconnaissance  Attack  Helicopter  (RAH-66) 

DDG  1000  Zumwalt  Class  Destroyer 
E-2D  Advanced  Hawkeye  (AHE) 

Evolved  Expendable  Launch  Vehicle 
Expeditionary  Fighting  Vehicle  (EFV) 

F/A-18E/F  Naval  Strike  Fighter  (SUPER  HORNET) 
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F-22  Raptor  Advanced  Tactical  Fighter  Aircraft 
F-35  Joint  Strike  Fighter  (JSF) 

Family  of  Medium  Tactical  Vehicles  (FMTV) 

Force  XXI  Battle  Command  Brigade  and  Below  (FBCB2) 

Guided  Multiple  Launch  Rocket  System  (GMLRS) 

H-l  Upgrades  (4BW/4BN) 

Increment  1  Early-Infantry  Brigade  Combat  Team 
Joint  Air-to-Surface  Standoff  Missile  (JASSM) 

Joint  Land  Attack  Cruise  Missile  Defense  Elevated  Netted  Sensor  System  (JLENS) 
Joint  Primary  Aircraft  Training  System  (JPATS) 

Joint  Tactical  Radio  System  (JTRS)  Ground  Mobile  Radio  (GMR) 

Land  Warrior  (LW) 

Littoral  Combat  Ship  (LCS) 

LPD  17  San  Antonio  Class  Amphibious  Transport  Dock 
MFI-60R  Multi-Mission  Helicopter 
MH-60S  Fleet  Combat  Support  Helicopter 

National  Polar-orbiting  Operational  Environmental  Satellite  System  (NPOESS) 

NAVSTAR  Global  Positioning  System 

Navy  Area  Theater  Ballistic  Missile  Defense  (TBMD) 

Remote  Minehunting  System 

RQ-4A/B  Global  Hawk  Unmanned  Aircraft  System 

Sense  and  Destroy  Armor  (SADARM) 

Space  Based  Infrared  System  (SBIRS)  High  Program 
SSN  774  Virginia  Class  Submarine 
V-22  Osprey  Joint  Services  Advanced  Vertical  Lift  Aircraft 
Warfighter  Information  Network — Tactical  (WIN-T) 

Wideband  Global  SATCOM  (WGS) 

XM982  155mm  Precision  Guided  Extended  Range  Artillery  Projectile  (Excalibur) 
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This  report  presents  the  results  of  two  studies:  The  first  compares  the  capabilities 
and  development  approaches  used  in  the  Joint  Tactical  Radio  System  wideband 
networking  waveform  (WNW)  and  the  commercial  long-term  evolution 
waveform,  and  the  second  analyzes  military  acquisition  programs  that  have 
repeatedly  exceeded  certain  cost  thresholds.  The  first  study  compares  differences 
in  system  designs,  technical  requirements,  intellectual  property  protection 
schemes,  and  cost  in  the  development  of  WNW.  It  also  examined  how  technical 
risks  and  challenging  requirements  contributed  to  schedule  and  cost  increases. 
The  second  study  attempts  to  identify  unique  characteristics  of  programs  that 
overrun  their  budgets  more  than  once. 
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